Sorting
From A to Z
Deployments found: 3
CHALLENGE
Bank of Hope needed a way to easily investigate potentially risky IPs without having to log in to multiple security product dashboards. The bank depends on its security information and event management (SIEM) tool as the heart of its incident response program, but when the SIEM flagged a potential problem IP address the analysts needed to spend up to a half hour confirming its reputation.
SOLUTION
ThreatStream offered Bank of Hope a way to sync its actionable intelligence with the organization’s SIEM tool and provide analysis with minimal effort.
RESULTS
• Reduced Mean-Time-To-Know
• SIEM Integration
• Headcount Savings
BANK OF HOPE CHALLENGE When the SIEM pointed to a threat indication, IT security analysts spent an inordinate amount of time looking up potential malicious IPs to confirm their current reputation. Bank of Hope had several systems in its IT environment that provided outside threat intelligence related to malicious IPs, but each of these had its own portal and its own dashboards. Each system provided threat intelligence, but none were intuitively embedded with the SIEM.
So analysts were left with a manual process that required them to look up information within each IT tool that had its own built-in threat information. With a lean staff, the bank could ill afford the kind of resource drain that looking up suspicious IPs was putting on its security operations. Staffers could take up to a half hour simply to determine whether the IP address had a known bad reputation, let alone to start acting on a potential incident once bad news was confirmed.
“Doing the research was a strenuous process,” said Arindam Bose, senior vice president and security officer for Bank of Hope. “We had to go to multiple resources to understand the indication of relevance of that IP address to our environment.”
Bank of Hope needed a way to simplify the process so it could make better use of its analysts’ bandwidth to work deeper into the forensics and incident response process.
OVERVIEW
Operating with $7.3 billion in assets Bank of Hope is the largest KoreanAmerican bank in the nation. As a major community financial institution with 50 branches across the U.S., Bank of Hope understandably must protect itself from a range of attacks against its IT systems. To keep tabs on the numerous security controls and monitoring systems it has in place, the bank depends on its security information and event management (SIEM) system to correlate events and help its analysts stay on top of trends. Unfortunately, until recently the bank’s IT security analysts were taxed by the amount of work needed to analyze and verify indicators of compromise (IOCs) related to outside IP addresses that surfaced from its SIEM correlation engine.
THE THREATSTREAM SOLUTION
The bank turned to the power of ThreatStream to do exactly that. According to Bose, Bank of Hope chose ThreatStream for several reasons.
First and foremost, the ThreatStream Threat Intelligence Platform is able to tell analysts with just a few clicks what an IP address’ threat score is, along with the confidence level based on reputation ranking.
Not only is it able to utilize threat feeds already available to Bank of Hope, but it also provides other feeds that add value to Bank of Hope’s analyses. In addition to IP reputation analysis, the tool can also replay executables in its sandbox environment to give Bank of Hope analysts a leg up on early analysis of potential IOCs and threat indicators.
But most importantly, ThreatStream integrates into Bank of Hope’s SIEM, so staffers do not need to reroute their analysis process and can do early investigation from a single centralized platform.
“The SIEM is a critical component of our environment and the heart of our program. It pulls in logs from a variety of different systems and correlates those indications to determine whether an activity
is malicious or not,” Bose says. “Integrating ThreatStream in our SIEM portal means we don’t have to go into five different systems, but can look at the validity of an IP or executable from a single place. The solution has minimized much of the team’s overhead.”
In addition, the bank needed a tool that could work with the FS-ISAC threat intelligence feed for information specific to the financial industry.
ThreatStream worked with the bank to develop that capability natively. It was this last point that truly tipped the scale in favor of ThreatStream for Bank of Hope.
Deployment was relatively painless for Bank of Hope, only requiring about an hour a week for the first month. The institution credits ThreatStream’s team with offering lots of guidance to get off the
ground running.
THE THREATSTREAM IMPACT
Now that the tool is in place, Bose reports the value of ThreatStream to Bank of Hope is in the time it saves analysts and the opportunity they have to address more threats than they once could.
The time it takes to analyze a threat has gone down from 30 minutes to just a few minutes, time that adds up over the course of investigating many malicious IPs every week. “There has been a substantial decrease in terms of meantime-toknow,” Bose says.
These efficiencies have enabled Bank of Hope to save on headcount. Because the tool automatically handles a large analytical workload, Bank of Hope was able to increase capacity without having to hire one or two additional analysts. What’s more, the false positive rates have been very low, meaning analysts spend very little time chasing non-existent problems.
Overall, the ThreatStream implementation has been a huge success for the Bank of Hope team, so much so that it is now looking at integrating the tool into its IDS/IPS, giving it the potential to automatically block threats with very high malicious confidence ratings.
SOLUTION
ThreatStream offered Bank of Hope a way to sync its actionable intelligence with the organization’s SIEM tool and provide analysis with minimal effort.
RESULTS
• Reduced Mean-Time-To-Know
• SIEM Integration
• Headcount Savings
BANK OF HOPE CHALLENGE When the SIEM pointed to a threat indication, IT security analysts spent an inordinate amount of time looking up potential malicious IPs to confirm their current reputation. Bank of Hope had several systems in its IT environment that provided outside threat intelligence related to malicious IPs, but each of these had its own portal and its own dashboards. Each system provided threat intelligence, but none were intuitively embedded with the SIEM.
So analysts were left with a manual process that required them to look up information within each IT tool that had its own built-in threat information. With a lean staff, the bank could ill afford the kind of resource drain that looking up suspicious IPs was putting on its security operations. Staffers could take up to a half hour simply to determine whether the IP address had a known bad reputation, let alone to start acting on a potential incident once bad news was confirmed.
“Doing the research was a strenuous process,” said Arindam Bose, senior vice president and security officer for Bank of Hope. “We had to go to multiple resources to understand the indication of relevance of that IP address to our environment.”
Bank of Hope needed a way to simplify the process so it could make better use of its analysts’ bandwidth to work deeper into the forensics and incident response process.
OVERVIEW
Operating with $7.3 billion in assets Bank of Hope is the largest KoreanAmerican bank in the nation. As a major community financial institution with 50 branches across the U.S., Bank of Hope understandably must protect itself from a range of attacks against its IT systems. To keep tabs on the numerous security controls and monitoring systems it has in place, the bank depends on its security information and event management (SIEM) system to correlate events and help its analysts stay on top of trends. Unfortunately, until recently the bank’s IT security analysts were taxed by the amount of work needed to analyze and verify indicators of compromise (IOCs) related to outside IP addresses that surfaced from its SIEM correlation engine.
THE THREATSTREAM SOLUTION
The bank turned to the power of ThreatStream to do exactly that. According to Bose, Bank of Hope chose ThreatStream for several reasons.
First and foremost, the ThreatStream Threat Intelligence Platform is able to tell analysts with just a few clicks what an IP address’ threat score is, along with the confidence level based on reputation ranking.
Not only is it able to utilize threat feeds already available to Bank of Hope, but it also provides other feeds that add value to Bank of Hope’s analyses. In addition to IP reputation analysis, the tool can also replay executables in its sandbox environment to give Bank of Hope analysts a leg up on early analysis of potential IOCs and threat indicators.
But most importantly, ThreatStream integrates into Bank of Hope’s SIEM, so staffers do not need to reroute their analysis process and can do early investigation from a single centralized platform.
“The SIEM is a critical component of our environment and the heart of our program. It pulls in logs from a variety of different systems and correlates those indications to determine whether an activity
is malicious or not,” Bose says. “Integrating ThreatStream in our SIEM portal means we don’t have to go into five different systems, but can look at the validity of an IP or executable from a single place. The solution has minimized much of the team’s overhead.”
In addition, the bank needed a tool that could work with the FS-ISAC threat intelligence feed for information specific to the financial industry.
ThreatStream worked with the bank to develop that capability natively. It was this last point that truly tipped the scale in favor of ThreatStream for Bank of Hope.
Deployment was relatively painless for Bank of Hope, only requiring about an hour a week for the first month. The institution credits ThreatStream’s team with offering lots of guidance to get off the
ground running.
THE THREATSTREAM IMPACT
Now that the tool is in place, Bose reports the value of ThreatStream to Bank of Hope is in the time it saves analysts and the opportunity they have to address more threats than they once could.
The time it takes to analyze a threat has gone down from 30 minutes to just a few minutes, time that adds up over the course of investigating many malicious IPs every week. “There has been a substantial decrease in terms of meantime-toknow,” Bose says.
These efficiencies have enabled Bank of Hope to save on headcount. Because the tool automatically handles a large analytical workload, Bank of Hope was able to increase capacity without having to hire one or two additional analysts. What’s more, the false positive rates have been very low, meaning analysts spend very little time chasing non-existent problems.
Overall, the ThreatStream implementation has been a huge success for the Bank of Hope team, so much so that it is now looking at integrating the tool into its IDS/IPS, giving it the potential to automatically block threats with very high malicious confidence ratings.
CHALLENGE
Blackhawk Network Holdings threat intelligence was the result of a combination of tools pieced together, none of which were integrated with their SIEM implementation, or provided enough context around IOCs to understand their potential impact.
Blackhawk Network Holdings needed a way to easily investigate potentially risky alerts without having to log in to multiple security product dashboards, reduce their manual overhead requirements, and maximize their resources so their analysts could better focus on critical issues. SOLUTION
Anomali® ThreatStream® offered Blackhawk Network Holdings a way to sync actionable threat intelligence with their SIEM alerts, integrate disparate threat feeds into one single-view dashboard, and
provide the context around IOCs necessary to understand their true importance. RESULTS
• Single dashboard and consolidation of all threat intelligence feeds
• Seamless SIEM integration
• Sandboxed testing environment to detonate payloads
• Improved threat analysis and response times
• More efficient and effective workflow
• Reduced false positives by over 95%
Before Anomali, Blackhawk Network Holdings relied on a variety of different security tools to manage their threat intelligence—a task they found extremely challenging. Like many organizations, they leveraged their security information and event management (SIEM) system to correlate events and help their analysts stay on top of trends. The problem was they had several systems in their IT environment that provided outside threat intelligence, each with its own portal and own dashboards. None of the systems integrated directly with their SIEM or communicated with each other. And the information was often duplicated or even worse, in disagreement. That meant whenever their SIEM pointed to a threat indication, their security analysts had to spend an inordinate amount of time analyzing and verifying indicators of compromise (IOCs) related to outside IP addresses. Thousands of alerts a day were more than the team could manage, let alone respond to.
Blackhawk Network Holdings wanted to simplify their threat intelligence processes so their analysts could focus more on forensics and remediation and less on research, management, and manual correlation. And they wanted to understand not just the type of attacks they were seeing, but the context of who their attackers were. They wanted a tool that could move their security forward but could also integrate with their current processes.
THE ANOMALI SOLUTION Blackhawk Network Holdings deployed Anomali ThreatStream, giving them an immediate threat intelligence solution via four key benefits: 1. Consolidation:
ThreatStream consolidated all of Blackhawk Network Holdings’ sources of threat information into one dashboard view within their SIEM, reducing duplicated information and false positives. In turn, they were able to minimize much of their security team’s manual overhead, allowing them to focus on resolution and not research. 2. Integration:
ThreatStream integrates directly into Blackhawk Network Holdings’ SIEM, so analysts do not need to reroute their analysis process and can do their early investigation from there. 3. Correlation: ThreatStream gave Blackhawk Network Holdings a way to correlate actionable threat intelligence SIEM alerts within their SIEM. ThreatStream tells analysts the threat score for each IP address, along with the confidence level based on a reputation ranking of its maliciousness.
"Unless we know who is after us, alerts lack context without Anomali" – Devin Ertel, CISO, Blackhawk Network Holdings.
4. Detonation:
ThreatStream enables analysts to replay executables in a sandboxed environment, giving them a safe place to test and a way to perform early analysis of potential IOCs and threat indicators.
"When a suspicious email comes in, we can detonate it in a sandboxed environment to see if it’s a threat. We couldn’t do that before". – Pablo Vega, Principal Security Engineer, Blackhawk Network Holdings
“Before Anomali, we had tons of information without context. We had to look through thousands of alerts quickly just to see what stood out and then react to those.” – Devin Ertel, CISO, Blackhawk Network Holdings
THE ANOMALI IMPACT ThreatStream gave Blackhawk Network Holdings the key capabilities and threat intelligence context that allowed their analysts to shift from searching through emails and dashboards to verify alerts to focusing on critical threats and issues. With ThreatStream, Blackhawk Network Holdings has higher confidence that critical alerts are malicious and not false positives. ThreatStream has provided them with greater visibility into what threats they confront. And since false positives have been very low in both number and criticality, analysts have been spending less time chasing non-existent problems and more time focusing on solutions. The value of ThreatStream is in the time it saves analysts and the opportunity they have to address more threats than they once could. Because the tool automatically handles a large analytical workload, Blackhawk Network Holdings was able to increase capacity without having to hire additional staff. ThreatStream has been an incredible solution for Blackhawk Network Holdings, allowing them to maximize resources and focus on the threats that matter most. ThreatStream gives Blackhawk Network Holding the ability to curate and filter the information they need from all of their sources of threat intel. And they’ve been able to apply ThreatStream security context around their alerts, helping to separate the high priority threat intel from low priority alerts to improve their overall security posture.
LONG TERM SUCCESS Blackhawk Network Holdings is now looking at integrating Anomali ThreatStream intelligence context into more internal security tooling, giving them the potential to automatically respond to threats with very high malicious confidence ratings. Blackhawk Network Holdings is interested in expanding their capabilities with Anomali Match™ and Anomali Lens™.
Blackhawk Network Holdings needed a way to easily investigate potentially risky alerts without having to log in to multiple security product dashboards, reduce their manual overhead requirements, and maximize their resources so their analysts could better focus on critical issues. SOLUTION
Anomali® ThreatStream® offered Blackhawk Network Holdings a way to sync actionable threat intelligence with their SIEM alerts, integrate disparate threat feeds into one single-view dashboard, and
provide the context around IOCs necessary to understand their true importance. RESULTS
• Single dashboard and consolidation of all threat intelligence feeds
• Seamless SIEM integration
• Sandboxed testing environment to detonate payloads
• Improved threat analysis and response times
• More efficient and effective workflow
• Reduced false positives by over 95%
Before Anomali, Blackhawk Network Holdings relied on a variety of different security tools to manage their threat intelligence—a task they found extremely challenging. Like many organizations, they leveraged their security information and event management (SIEM) system to correlate events and help their analysts stay on top of trends. The problem was they had several systems in their IT environment that provided outside threat intelligence, each with its own portal and own dashboards. None of the systems integrated directly with their SIEM or communicated with each other. And the information was often duplicated or even worse, in disagreement. That meant whenever their SIEM pointed to a threat indication, their security analysts had to spend an inordinate amount of time analyzing and verifying indicators of compromise (IOCs) related to outside IP addresses. Thousands of alerts a day were more than the team could manage, let alone respond to.
Blackhawk Network Holdings wanted to simplify their threat intelligence processes so their analysts could focus more on forensics and remediation and less on research, management, and manual correlation. And they wanted to understand not just the type of attacks they were seeing, but the context of who their attackers were. They wanted a tool that could move their security forward but could also integrate with their current processes.
THE ANOMALI SOLUTION Blackhawk Network Holdings deployed Anomali ThreatStream, giving them an immediate threat intelligence solution via four key benefits: 1. Consolidation:
ThreatStream consolidated all of Blackhawk Network Holdings’ sources of threat information into one dashboard view within their SIEM, reducing duplicated information and false positives. In turn, they were able to minimize much of their security team’s manual overhead, allowing them to focus on resolution and not research. 2. Integration:
ThreatStream integrates directly into Blackhawk Network Holdings’ SIEM, so analysts do not need to reroute their analysis process and can do their early investigation from there. 3. Correlation: ThreatStream gave Blackhawk Network Holdings a way to correlate actionable threat intelligence SIEM alerts within their SIEM. ThreatStream tells analysts the threat score for each IP address, along with the confidence level based on a reputation ranking of its maliciousness.
"Unless we know who is after us, alerts lack context without Anomali" – Devin Ertel, CISO, Blackhawk Network Holdings.
4. Detonation:
ThreatStream enables analysts to replay executables in a sandboxed environment, giving them a safe place to test and a way to perform early analysis of potential IOCs and threat indicators.
"When a suspicious email comes in, we can detonate it in a sandboxed environment to see if it’s a threat. We couldn’t do that before". – Pablo Vega, Principal Security Engineer, Blackhawk Network Holdings
“Before Anomali, we had tons of information without context. We had to look through thousands of alerts quickly just to see what stood out and then react to those.” – Devin Ertel, CISO, Blackhawk Network Holdings
THE ANOMALI IMPACT ThreatStream gave Blackhawk Network Holdings the key capabilities and threat intelligence context that allowed their analysts to shift from searching through emails and dashboards to verify alerts to focusing on critical threats and issues. With ThreatStream, Blackhawk Network Holdings has higher confidence that critical alerts are malicious and not false positives. ThreatStream has provided them with greater visibility into what threats they confront. And since false positives have been very low in both number and criticality, analysts have been spending less time chasing non-existent problems and more time focusing on solutions. The value of ThreatStream is in the time it saves analysts and the opportunity they have to address more threats than they once could. Because the tool automatically handles a large analytical workload, Blackhawk Network Holdings was able to increase capacity without having to hire additional staff. ThreatStream has been an incredible solution for Blackhawk Network Holdings, allowing them to maximize resources and focus on the threats that matter most. ThreatStream gives Blackhawk Network Holding the ability to curate and filter the information they need from all of their sources of threat intel. And they’ve been able to apply ThreatStream security context around their alerts, helping to separate the high priority threat intel from low priority alerts to improve their overall security posture.
LONG TERM SUCCESS Blackhawk Network Holdings is now looking at integrating Anomali ThreatStream intelligence context into more internal security tooling, giving them the potential to automatically respond to threats with very high malicious confidence ratings. Blackhawk Network Holdings is interested in expanding their capabilities with Anomali Match™ and Anomali Lens™.
ABOUT FEDERAL SYSTEMS INTEGRATOR
This Federal Systems Integrator (FSI) is a proven provider of information solutions, engineering and analytics for the U.S. Intelligence Community, U.S. Department of Defense and other federal agencies. With more than 40 years of experience, this FSI designs, develops and delivers high impact, mission-critical services and solutions to overcome it’s customers’ most complex problems.
THE PROBLEM
Working primarily as a systems integrator with clients in sensitive intelligence and security communities, this FSI’s intellectual property (IP) contains critical high-value information. This IP, essential to the U.S. government, must remain protected and secure.
On a daily basis, this FSI receives hundreds of Indicators of Compromise (IOCs) from multiple sources, and each IOC requires evaluation of the level of confidence behind the intelligence. Analysis of the data must:
This FSI needed a way to speed threat intelligence validation and integration, and to do it without compromising information security. The company sought an automated threat intelligence solution that would work with this FSI’s existing security information event management (SIEM) tools while reducing the time spent analyzing and operationalizing threat intelligence data.
THE THREATSTREAM SOLUTION
This FSI turned to ThreatStream for an automated cyber threat intelligence solution. The ThreatStream Optic™ platform counters adversaries by fusing actionable intelligence with existing security infrastructure by:
Before ThreatStream, this FSI staff spent thousands of hours annually to collect intelligence, sift through IOCs, validate intelligence and then operationalize that data by writing rules and actions into security infrastructure.
This FSI deployed ThreatStream Optic and immediately reduced the amount of time it took to not only identify valid threat intelligence, but also operationalize that threat intel by injecting it directly into this FSI’s existing security tools. ThreatStream Optic connects with this FSI’s SIEM through a single, cloud-based portal, consolidating, normalizing and validating intelligence.
This seamless integration also eliminates the time and resource-intensive process of manually de-duplicating information from multiple feeds.
This FSI chose ThreatStream because the ThreatStream Optic platform, unlike other threat feeds, provides the additional benefit of cross-validation analysis. This FSI is able to take the threat intel received from ThreatStream and other sources and use ThreatStream Optic to determine with a high degree of probability what is valid intelligence, and act accordingly. ThreatStream allows this FSI to act on threat intel with a high degree of confidence.
The efficiencies created by ThreatStream Optic also allow this FSI to redeploy valuable human resources, which saves this FSI countless hours and thousands of dollars per year.
“Rather than taking us days to implement threat intelligence into our cybersecurity tools, with Optic, we can do it in minutes.”
IMPLEMENTATION
ThreatStream provided this FSI integrations for multiple sets of technology architecture, ensuring a smooth implementation. This FSI’s SIEM tools easily connect with ThreatStream’s server to pull down and inject data directly into this FSI’s security architecture stack. The threat intelligence provided by ThreatStream is viewed and used at this FSI’s highest levels.
“The reliability of the data and depth of information the ThreatStream solution provides is top-notch. ThreatStream only delivers data that’s been fully vetted, rich with context and insights, allowing us to take immediate action.”
A PARTNERSHIP
“Working with ThreatStream is really a partnership. We have regularly scheduled discussions, and if we need anything, it’s only a phone call away. It’s easy to communicate with our ThreatStream team, and they are very receptive of what we ask of them.”
ThreatStream Optic is the first threat intelligence platform that manages the entire life cycle of threat intelligence from multi-source acquisition to operational integration across the entire ecosystem of existing security devices. ThreatStream Optic enables enterprise and government organizations to seamlessly aggregate and analyze threat intelligence and automatically inject the information into their security infrastructure.
THE PROBLEM
Working primarily as a systems integrator with clients in sensitive intelligence and security communities, this FSI’s intellectual property (IP) contains critical high-value information. This IP, essential to the U.S. government, must remain protected and secure.
On a daily basis, this FSI receives hundreds of Indicators of Compromise (IOCs) from multiple sources, and each IOC requires evaluation of the level of confidence behind the intelligence. Analysis of the data must:
- Consolidate important threat intel data
- Put the intel into context
- Decide if intel is pertinent and reliable
- Show where to focus and take action
This FSI needed a way to speed threat intelligence validation and integration, and to do it without compromising information security. The company sought an automated threat intelligence solution that would work with this FSI’s existing security information event management (SIEM) tools while reducing the time spent analyzing and operationalizing threat intelligence data.
THE THREATSTREAM SOLUTION
This FSI turned to ThreatStream for an automated cyber threat intelligence solution. The ThreatStream Optic™ platform counters adversaries by fusing actionable intelligence with existing security infrastructure by:
- Consolidating and curating multiple threat intelligence sources while eliminating redundancies
- Providing cross-validated analysis
- Rapidly operationalizing intelligence with high confidence
Before ThreatStream, this FSI staff spent thousands of hours annually to collect intelligence, sift through IOCs, validate intelligence and then operationalize that data by writing rules and actions into security infrastructure.
This FSI deployed ThreatStream Optic and immediately reduced the amount of time it took to not only identify valid threat intelligence, but also operationalize that threat intel by injecting it directly into this FSI’s existing security tools. ThreatStream Optic connects with this FSI’s SIEM through a single, cloud-based portal, consolidating, normalizing and validating intelligence.
This seamless integration also eliminates the time and resource-intensive process of manually de-duplicating information from multiple feeds.
This FSI chose ThreatStream because the ThreatStream Optic platform, unlike other threat feeds, provides the additional benefit of cross-validation analysis. This FSI is able to take the threat intel received from ThreatStream and other sources and use ThreatStream Optic to determine with a high degree of probability what is valid intelligence, and act accordingly. ThreatStream allows this FSI to act on threat intel with a high degree of confidence.
The efficiencies created by ThreatStream Optic also allow this FSI to redeploy valuable human resources, which saves this FSI countless hours and thousands of dollars per year.
“Rather than taking us days to implement threat intelligence into our cybersecurity tools, with Optic, we can do it in minutes.”
IMPLEMENTATION
ThreatStream provided this FSI integrations for multiple sets of technology architecture, ensuring a smooth implementation. This FSI’s SIEM tools easily connect with ThreatStream’s server to pull down and inject data directly into this FSI’s security architecture stack. The threat intelligence provided by ThreatStream is viewed and used at this FSI’s highest levels.
“The reliability of the data and depth of information the ThreatStream solution provides is top-notch. ThreatStream only delivers data that’s been fully vetted, rich with context and insights, allowing us to take immediate action.”
A PARTNERSHIP
“Working with ThreatStream is really a partnership. We have regularly scheduled discussions, and if we need anything, it’s only a phone call away. It’s easy to communicate with our ThreatStream team, and they are very receptive of what we ask of them.”
ThreatStream Optic is the first threat intelligence platform that manages the entire life cycle of threat intelligence from multi-source acquisition to operational integration across the entire ecosystem of existing security devices. ThreatStream Optic enables enterprise and government organizations to seamlessly aggregate and analyze threat intelligence and automatically inject the information into their security infrastructure.
The ROI4CIO Deployment Catalog is a database of software, hardware, and IT service implementations. Find implementations by vendor, supplier, user, business tasks, problems, status, filter by the presence of ROI and reference.