Custom Intelligence Feeds feature provides an ability to add custom cyber intelligence feeds into the Threat Prevention engine. It allows fetching feeds from a third-party server directly to the Security Gateway to be enforced by Anti-Virus and Anti-Bot blades.

The Custom Intelligence Feeds feature also assists customers with the operational and engineering management challenges they face handling indicators: Managing and monitoring of the custom intelligence feeds is done with minimal operational overhead.

Indicator is a pattern of relevant observable malicious activity in an operational cyber domain, with relevant information on how to interpret it and how to handle it. Indicators are derived from intelligence, self-analysis and/or governments, partners etc.

Observable is an event or a stateful property that can be observed in an operational cyber domain. For example: IP address, MD5 file signature, URL, Mail sender address.

Table of Contents

• Supported Formats
• Installation
• Fetching new feed using CLI - ioc_feeds
• Known feeds examples
• Known Limitations
• Troubleshooting

Supported Formats

Each indicator file can be either in CSV, SNORT or STIX XML format.

CSV (*.csv) format

• Delimiter between fields is ','
• Delimiter between records is '\n'
• Fields are plain text
• Each record "must" contain the same number of comma-separated fields
• Quoted fields are accepted.
• The first line/record, prefixed with #, is a header containing column names in each of the fields.

Example:

#! DESCRIPTION = indi file,,,,,,
"#! REFERENCE = Indicator Bulletin; Feb 20, 2014",,,,,,
# FILE FORMAT:,,,,,,
"#      All lines beginning  ""#"" are comments",,,,,,
"#      All lines beginning  ""#!"" are metadata read by the SW",,,,,,
"# UNIQ-NAME,VALUE,TYPE,CONFIDENCE,SEVERITY,PRODUCT,COMMENT",,,,,,
observ1,8d9b6b8912a2ed175b77acd40cbe9a73,MD5,medium,medium,AV,FILENAME:WUC Invitation Letter Guests.doc
observ2,76700f862a0c241b8f4b754f76957bda,MD5,high,high,AV,FILENAME:essais~.swf|
NOTE:FWS type Flash file
observ7,http://somemaliciousdomain.com/uploadfiles/upload/exp.swf?info=
789c333432d333b4d4b330d133b7b230b03000001b39033b&infosize=00840000 
,URL,high,high,AV,IPV4ADDR:196.168.25.25
observ8,svr01.passport.ServeUser.com,Domain,low,high,AB,TCP:80|
IPV4ADDR:172.18.18.25|NOTE:Embedded EXE Remote C&C and Encoded Data
observ9,somemaliciousdomain2.com,Domain,,low,AV,TCP:8080|IPV4ADDR:172.22.14.10
observ10,http://www.bogusdomain.com/search?q=%24%2B%25&form=MOZSBR&pc=
MOZI,URL,low,low,AB,IPV4ADDR:172.25.1.5
observ11,http://somebogussolution.com/register/card/log.asp?isnew=-1&LocalInfo=
Microsoft%20Windows%20XP%20Service%20Pack%202&szHostName=
ADAM-E512679EFD&tmp3=tmp3,URL,medium,,AB,
observ14,172.16.47.44,IP,high,medium,AB,TCP:8080
observ15,172.16.73.69,IP,medium,medium,AV,TCP:443|NOTE:Related to Flash 
exploitation
observ16,abc@def.com,mail-to,,high,AV,"NOTE:truncated; samples have appended to 
the subject the string ""PH000000NNNNNNN"" where NNNNNNN is a varying number"
observ34,stamdomain.com,domain,,,AB,
observ35,stamdomain.com,mail-from,high,medium,AV,
observ37,xyz.com,mail-from,medium,medium,AB,
observ38,@xyz.com,mail-from,medium,medium,AB,    

 

Custom CSV format

Custom Intelligence Feeds feature supports different kinds of CSV structure files.

For the fetch to succeed, you must define the file's format, delimiter, and the comment lines to skip.

Syntax Notes:

• The supported fields are: [name,value,type,confidence,severity,product,comment]

• To use a field value from the original file (must be one of the supported fields listed above), you must specify its location in the csv row by using #index.
  For example, if you want to take the 3rd index in your csv file to be your observable's comment in the new file, use:

   --format [comment:#3]

• To use a default value for all observables, use

--format [type:domain]

• The Value field is mandatory and must be taken from the original file.

• The Type field is mandatory and must be taken from the original file or be sent as a default value for all observables.

• All other fields are optional and can either be taken from the original file or sent as a default value for all observables.

• When the feed's resource is a remote source (transport equals HTTP or HTTPS) - every time the feed will be fetched, it will parse according to the format that has been specified for this feed.

 

Examples:

• Original CSV structure is a list of IP addresses

  ioc_feeds add --feed_name ip_list --transport http --resource "http://blocklist.greensnow.co/greensnow.txt" --format [value:1,type:ip]

  

• Original CSV structure is a list of Domains

  ioc_feeds add --feed_name domain_list --transport https --resource "https://urlhaus.abuse.ch/downloads/text/" --format [type:domain,value:1] --comment "#, Site"

  

• Original CSV structure is a list of IP addresses in CIDR format and comment lines are marked as ';'

  ioc_feeds add --feed_name ip_cidr_list_with_delimiter --transport https --resource "https://www.spamhaus.org/drop/edrop.txt" --format [value:1,type:ip] --delimiter ";" --comment ";"

  
  

• Original CSV structure is a list of IP addresses separated by '|' delimiter, and comment lines are marked as '#'

  ioc_feeds add --feed_name ip_list_with_spaces --transport local_file --resource "/home/admin/ioc/ip_list_with_spaces.txt" --format [value:3,comment:#2,type:ip] --comment [#] --delimiter "|"

  

• Original CSV structure is a list of different types separated by ',' delimiter, and comment lines are marked as '#' or 'Site'

  ioc_feeds add --feed_name try_custom_csv --transport http --resource http://192.168.13.13/ioc/bad_csv_format.csv --format [type:#1,value:#3,name:#6,comment:#7,product:av] --comment [#, Site] --delimiter ,

  

• Original CSV structure is a list of different types separated by ',' delimiter

  ioc_feeds add --feed_name ioc_feed --transport http --resource "http://www.malwaredomainlist.com/updatecsv.php" --format [value:3,comment:#2,type:ip] --delimiter ,

  

STIX - Structured Threat Information eXpression

STIX™ is a collaborative community-driven effort to define and develop a standardized language to represent structured cyber threat information.
For more information, refer to STIX 1.x Archive Website.

Supported observables types

• URL
• Domain
• IP
• IP Range
• MD5
• Mail-subject
• Mail-from
• Mail-to
• Mail-cc
• Mail-reply-to
• SHA1, (from R80.40)
• SHA256, (from R80.40)

Check Point Format

Check Point format has the following structure:

#UNIQ-NAME,VALUE,TYPE,CONFIDENCE,SEVERITY,PRODUCT,COMMENT
observ1,www.websitename.com,URL,medium,low,AV,Website articles Accessobserv2,192.168.38.187,IP,high,high,AB,C&C server IP
observ4,b04552c6ede6c7f47327f304a6ac1a20,MD5,low,high,AV,Security.doc

Snort Format

Read more about Snort format feed here

Installation

The feature is integrated in version R80.30 and higher.

Note: To import external Custom Intelligence Feeds using SmartConsole in versions R81 and higher, refer to: Threat Prevention R81 Administration Guide > Configuring Advanced Threat Prevention Settings > Configuring Threat Indicators > Importing External Custom Intelligence Feeds > Importing External Custom Intelligence Feeds in SmartConsole.

Fetching new feed using CLI - ioc_feeds

Note: Starting from R81, the ioc_feeds command also supports the "--yes" command line argument, which when specified, avoids any user interactions with yes/no questions and assume the user answers 'yes'. This option is hidden and is not visible in the command usage output. This allows the command to be scripted for automation.

Feed's resource can be:

• URL - HTTP/HTTPS (--transport http --resource "http://10.0.0.1/my_feeds/stix_feed.xml")
  *Self-signed certificate HTTPS resource will prompt for user agreement to update the bundle. It is possible to skip the certificate verification by running "export EXT_IOC_NO_SSL_VALIDATION=1" on the gateway but this does not survive active SSH session.
  • To add Environmental variable permanently on gateway,

• File on the machine (--transport local_file --resource "/home/admin/my_feed.csv")

• Directory on the machine, which contains the same feed_format - (--transport local_directory --resource "/home/admin/my_feed_folder")
  
  

Parameter	Description	Example
push	Push feeds now	ioc_feeds push
show	Prints all existing feeds	ioc_feeds show
show --feed_name <feed>	Prints specific feed details	ioc_feeds show --feed_name local_feed
show_interval	Prints fetching interval	ioc_feeds show_interval
set_interval sec	Set interval for fetching in seconds *Feed fetching interval - same for all feeds	ioc_feeds set_interval 1000
show_scanning_mode	Prints scanning mode	ioc_feeds show_scanning_mode
set_scanning_mode	Set scanning mode - on/off	ioc_feeds set_scanning_mode off
add	Adding new feed. Mandatory fields: --feed_name <feed> --transport <http/https/local_file/local_directory> --resource <url/full_path> Optional fields: --state <true/false> (active/inactive. default - True) --feed_action <Prevent/Detect/Ask> (default – Prevent) --user_name <user> (prompt for password) --proxy <proxy:port> --proxy <none> – don't use proxy --proxy <proxy:port> - override gatewayproxy for feed (not mentioning proxy flag - gateway proxy will be used) --proxy_user_name <user> (prompt for password) --test true test feed fetching and parsing	Examples: ioc_feeds add --feed_name local_feed --transport local_file --resource /home/admin/my_feed.csv ioc_feeds add --feed_name remote_feed --transport http --resource 10.0.0.1/my_feeds/stix_feed.xml --proxy 127.10.10.1:8080 --state false –feed_action detect --user_name admin@checkpoint.com
modify	Modify existing feed. Fields that will not be mentioned will stay as they were before	ioc_feeds modify --feed_name local_feed --state true
delete	Delete the existing feed	ioc_feeds delete --feed_name local_feed

 

More examples:

• Adding a new remote feed

[Expert@HostName:0]# ioc_feeds add --feed_name remote_csv_feed --transport http --resource "http://10.10.1.100/ioc/ioc_csv_file.csv" --feed_action Prevent

• Adding a new local feed

[Expert@HostName:0]# ioc_feeds add --feed_name ioc_stix_file --transport local_file --resource "/home/admin/ioc/ioc_stix_file.xml"

Note for R80.20SP:

Local feed content is refreshed periodically based on the feed fetching interval (see “set_interval” above).
When you modify the local feed source file, you must distribute the updated version with “asg_cp2blades”.
# vi /home/admin/ioc/ioc_stix_file.xml
# asg_cp2blades /home/admin/ioc/ioc_stix_file.xml

• Printing existing feeds

[Expert@HostName:0]# ioc_feeds show

• Deleting a feed

[Expert@HostName:0]# ioc_feeds delete --feed_name ioc_stix_file

• Test feed fetching and parsing

[Expert@HostName:0]# ioc_feeds add --feed_name remote_stix_file --transport http --resource "http://www.public_indicators.com/ioc_stix_file.xml" --test true

 

Known feeds examples (using the Custom CSV feature)

Description	URL	Command Line
Alienvault IP Reputation	http://reputation.alienvault.com/reputation.data	ioc_feeds add --feed_name reputation --transport http --resource "http://reputation.alienvault.com/reputation.data" --format [type:ip,value:#1,comment:#4] --delimiter "#"
Domains	https://www.botvrij.eu/data/ioclist.hostname.raw	ioc_feeds add --feed_name domains --transport https --resource "https://www.botvrij.eu/data/ioclist.hostname.raw" --format [type:domain,value:#1]
IPs	https://sslbl.abuse.ch/blacklist/sslipblacklist.csv	ioc_feeds add --feed_name ips --transport https --resource "https://sslbl.abuse.ch/blacklist/sslipblacklist.csv" --format [type:ip,value:#2] --comment [#] --delimiter ","
Talos IP Blacklist	http://www.talosintelligence.com/documents/ip-blacklist	ioc_feeds add --feed_name ip_blacklist --transport https --resource "https://www.talosintelligence.com/documents/ip-blacklist" --format [type:ip,value:#1]
Spam List	http://www.ipspamlist.com/public_feeds.csv	ioc_feeds add --feed_name spam_list --transport https --resource "https://www.ipspamlist.com/public_feeds.csv" --format [type:ip,value:#3,comment:#4] --comment ["#", "first_seen"] --delimiter ","
Cybercrime hash list	http://cybercrime-tracker.net/ccamlist.php	ioc_feeds add --feed_name hash_list --transport http --resource "http://cybercrime-tracker.net/ccamlist.php" --format [type:sha1,value:#1]

 

Managing Custom Intelligence Feeds from the Security Management Server

• In R80.30/R80.40 you can distribute the IOC feeds from the Management server with this commands:

  1. Run on one of your Security Gateways: 
     ioc_feeds export
  2. If the feeds were fetched successfully, move /home/admin/export_ioc.tar.gz file to the Security Management.
  3. From the GUI Command-line tool, run:
     put-file file-path "/home/admin" file-name "export_ioc.tar.gz" file-content @"/home/admin/export_ioc.tar.gz" --treat-value-as-file-by-prefix @ targets.1 "corporate-gateway" targets.2 "Backup-GW"
  4. From the GUI Command-line tool, run:
     run-script script-name "Import Custom feeds" script "ioc_feeds import /home/admin/export_ioc.tar.gz" targets.1 "corporate-gateway" targets.2 "Backup-GW"

• In R81 and higher versions refer to the Installation section above to manage Custom Intelligence Feeds in SmartConsole.

• Feed fetch interval can be configured via SmartConsole.
  Go to Manage & Settings -> Blades -> Threat Prevention -> Advanced Settings -> External Feed, Set new time and Install Policy
  * New interval will be set to all feeds
  
  

 

Known Limitations

• Observables of IP addresses and IP Ranges can hold IPv4 values only. In R81 and higher versions IPV6 is supported as well.

• MD5, SHA1, SHA256 observables cannot be enforced by Anti-Bot Blade. If user does not enable Anti-Virus blade, there will be no enforcement.

• For R80.20SP, a Jumbo Hotfix Accumulator installation is required.

• Inbound traffic to a host behind the gateway does not get blocked, e.g: IP that is on the feed, sends ICMP Request to a host behind the gateway. This traffic does not get blocked.

  In R81 and higher versions, this traffic is blocked.

• Not supported on version R81 SP

• Large feeds can take a lot of time to load on ext3 filesystem.

• From 81.20 - To prevent system overload feed won't be loaded if it exceeds 80% of total free disk space or 50% of free RAM.

• Before 81.20, there is limit of number of observables.

• ioc_feeds export is working only on R80.30/R80.40
• IP observables ignore the blade column in CSV format

 

Troubleshooting

• Run $FWDIR/bin/ioc_feeder -d -f on the Security Gateway to fetch feeds in debug mode. 
  Check $FWDIR/log/ioc_feeder.elg for debug output

• Verify that the $FWDIR/conf/ioc_feeder.conf configuration file exists and is not corrupt. If the file is corrupt, delete the feed and re-add with the proper changes

• Verify that there are no errors in these debug files:
  $FWDIR/log/ioc_feeder.elg
  $FWDIR/log/ext_ioc_push.elg
  
  
• For remote feeds (HTTP / HTTPS):
  • Web servers might return files with unauthorized or forbidden headers as a response. Make sure that file is fetched as expected and contains the correct information, whether it’s a STIX file or a CSV file, in: $FWDIR/external_ioc/feed_name_folder
  • For HTTPS remote feeds, if the certificate update process failed, you can skip the certificate verification. Run: export EXT_IOC_NO_SSL_VALIDATION=1 on the Security Gateway(does not survive particular SSH session).
    • To add Environmental variable permanently on gateway,

• • Check the Proxy configurations:
    • By default, the Security Gateway's proxy will be used
    • Override proxy configurations: --proxy <proxy:port>
    • Don't use proxy: --proxy <none>

Related Solution: sk181479 - Location of CSV Files on a Security Gateway

This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.