Support Center > Search Results > SecureKnowledge Details
What is "Custom Intelligence Feeds" feature?
Solution

Custom Intelligence Feeds feature provides an ability to add custom cyber intelligence feeds into the Threat Prevention engine. It allows fetching feeds from a third-party server directly to the Security Gateway to be enforced by Anti-Virus and Anti-Bot blades.

The Custom Intelligence Feeds feature also assists customers with the operational and engineering management challenges they face handling indicators: Managing and monitoring of the custom intelligence feeds is done with minimal operational overhead.

Indicator is a pattern of relevant observable malicious activity in an operational cyber domain, with relevant information on how to interpret it and how to handle it. Indicators are derived from intelligence, self-analysis and/or governments, partners etc.

Observable is an event or a stateful property that can be observed in an operational cyber domain. For example: IP address, MD5 file signature, URL, Mail sender address.

Supported Formats

Each indicator file can be either in CSV or STIX XML format.

CSV (*.csv) format

  • Delimiter between fields is ','
  • Delimiter between records is '\n'
  • Fields are plain text
  • Each record "must" contain the same number of comma-separated fields
  • Quoted fields are accepted.
  • The first line/record, prefixed with #, is a header containing column names in each of the fields.


Example:

#! DESCRIPTION = Example
#! REFERENCE = \\galaxy\all_users\mora\IOC\example.csv
# UNIQ-NAME,VALUE,TYPE,CONFIDENCE,SEVERITY,PRODUCT,COMMENT
observ1,1.2.3.4,IP,high,high,AB,C&C server IP
observ2,www.boredpanda.com,Domain,high,low,AV,BoredPanda Access
observ3,www.ynet.co.il/articles/,URL,medium,low,AV,Ynet articles Access
observ4,9364a29363851d53608334392cddab46,MD5,low,high,AV,Guests.doc

Custom CSV format

Custom Intelligence Feeds feature supports different kinds of CSV structure files.

For the fetch to succeed, you must define the file's format, delimiter, and the comment lines to skip.

Syntax Notes:

  • The supported fields are: [name,value,type,confidence,severity,product,comment]
  • To use a field value from the original file (must be one of the supported fields listed above), you must specify its location in the csv row by using #index.
    For example, if you want to take the 3rd index in your csv file to be your observable's comment in the new file, use:

   --format [comment:#3]

  • To use a default value for all observables, use

--format [type:domain]

  • The Value field is mandatory and must be taken from the original file.
  • The Type field is mandatory and must be taken from the original file or be sent as a default value for all observables.
  • All other fields are optional and can either be taken from the original file or sent as a default value for all observables.
  • When the feed's resource is a remote source (transport equals HTTP or HTTPS) - every time the feed will be fetched, it will parse according to the format that has been specified for this feed.

 

Examples:

  • Original CSV structure is a list of IP addresses

ioc_feeds add --feed_name ip_list --transport http --resource "http://blocklist.greensnow.co/greensnow.txt" --format [value:1,type:ip]


  • Original CSV structure is a list of Domains

ioc_feeds add --feed_name domain_list --transport https --resource "https://isc.sans.edu/feeds/suspiciousdomains_High.txt" --format [type:domain,value:1] --comment "#, Site"


  • Original CSV structure is a list of IP addresses in CIDR format and comment lines are marked as ';'

ioc_feeds add --feed_name ip_cidr_list_with_delimiter --transport https --resource "https://www.spamhaus.org/drop/edrop.txt" --format [value:1,type:ip] --delimiter ";" --comment ";"


  • Original CSV structure is a list of IP addresses seperated by '|' delimiter, and comment lines are marked as '#'

ioc_feeds add --feed_name ip_list_with_spaces --transport local_file --resource "/home/admin/ioc/ip_list_with_spaces.txt" --format [value:3,comment:#2,type:ip] --comment [#] --delimiter "|"


  • Original CSV structure is a list of different types seperated by ',' delimiter, and comment lines are marked as '#' or 'Site'

ioc_feeds add --feed_name try_custom_csv --transport http --resource http://192.168.13.13/ioc/bad_csv_format.csv --format [type:#1,value:#3,name:#6,comment:#7,product:av] --comment [#, Site] --delimiter ,


  • Original CSV structure is a list of different types seperated by ',' delimiter

ioc_feeds add --feed_name ioc_feed --transport http --resource "http://www.malwaredomainlist.com/updatecsv.php" --format [value:3,comment:#2,type:ip] --delimiter ,


STIX - Structured Threat Information eXpression

STIX™ is a collaborative community-driven effort to define and develop a standardized language to represent structured cyber threat information.
For more information, refer to STIX 1.x Archive Website.

Supported observables types

  • URL
  • Domain
  • IP
  • IP Range
  • MD5
  • Mail-subject
  • Mail-from
  • Mail-to
  • Mail-cc
  • Mail-reply-to


Installation

You may download the package of R80.10 or R80.20 for offline installation.

CPUSE package is available from the below table:

Product
CPUSE Online Identifier
Hotfix on top of R80.10 Jumbo HFA Take 121 Check_Point_R80.10_JHF_T121_Hotfix_sk132193_FULL.tgz
Hotfix on top of R80.20 GA (Take 101) Check_Point_R80.20_T101_Hotfix_sk132193_FULL.tgz

 

Fetching new feed using CLI - ioc_feeds

Feed's resource can be:

  • URL - HTTP/HTTPS (--transport http --resource "http://10.0.0.1/my_feeds/stix_feed.xml")
    *Self-signed certificate HTTPS resource will propmt for user agreement to update the bundle. It is possible to skip the certificate verification by running "export EXT_IOC_NO_SSL_VALIDATION=1" on the gateway.
  • File on the machine (--transport local_file --resource "/home/admin/my_feed.csv")
  • Directory on the machine, which contains the same feed_format - (--transport local_directory --resource "/home/admin/my_feed_folder")
Parameter Description Example
push push feeds now ioc_feeds push
show prints all existing feeds ioc_feeds show
show --feed_name <feed>  prints specific feed details  ioc_feeds show --feed_name local_feed
show_interval  prints fetching interval  ioc_feeds show_interval 
set_interval sec

set interval for fetching in seconds

*Feed fetching interval - same for all feeds

ioc_feeds set_interval 1000 
show_scanning_mode  prints scanning mode  ioc_feeds show_scanning_mode 
set_scanning_mode  set scanning mode - on/off  ioc_feeds set_scanning_mode off
add 

adding new feed

Mandatory fields:

--feed_name <feed>

--transport <http/https/local_file/local_directory>

--resource <url/full_path>

Optional fields:

--state <true/false> (active/inactive. default - True)

--feed_action <Prevent/Detect/Ask> (default – Prevent)

--user_name <user> (prompt for password)

--proxy <proxy:port>

--proxy <none> – don't use proxy

--proxy <proxy:port> - override gatewayproxy for feed

(not mentioning proxy flag - gateway proxy will be used)

--proxy_user_name <user> (prompt for password)

--test true
test feed fetching and parsing

Examples:

ioc_feeds add --feed_name local_feed --transport local_file --resource /home/admin/my_feed.csv

ioc_feeds add --feed_name remote_feed --transport http --resource 10.0.0.1/my_feeds/stix_feed.xml --proxy 127.10.10.1:8080 --state false –feed_action detect --user_name admin@checkpoint.com

modify 

modify existing feed

fields that will not be mentioned will stay as they were before 

ioc_feeds modify --feed_name local_feed --state true
delete  delete existing feed  ioc_feeds delete --feed_name local_feed 

 

More examples:

  • Adding a new remote feed

[Expert@HostName:0]# ioc_feeds add --feed_name remote_csv_feed --transport http --resource "http://10.10.1.100/ioc/ioc_csv_file.csv" --feed_action Prevent

  • Adding a new local feed

[Expert@HostName:0]# ioc_feeds add --feed_name ioc_stix_file --transport local_file --resource "/home/admin/ioc/ioc_stix_file.xml"

  • Printing existing feeds

[Expert@HostName:0]# ioc_feeds show

  • Deleting a feed

[Expert@HostName:0]# ioc_feeds delete --feed_name ioc_stix_file

  • Test feed fetching and parsing

[Expert@HostName:0]# ioc_feeds add --feed_name remote_stix_file --transport http --resource "http://www.public_indicators.com/ioc_stix_file.xml" --test true

 

Current Limitations

  • Observables of IP addresses and IP Ranges can hold IPv4 values only.
  • MD5 observables cannot be enforced by Anti-Bot Blade. If user does not enable Anti-Virus blade, there will be no enforcement.

 

Troubleshooting

  • Run $FWDIR/bin/ioc_feeder -d -f on the Security Gateway to fetch feeds in debug mode.

  • Verify that the $FWDIR/conf/ioc_feeder.conf configuration file exists and is not corrupt. If the file is corrupt, delete the feed and re-add with the proper changes
  • Verify that there are no errors in these debug files:
    $FWDIR/log/ioc_feeder.elg
    $FWDIR/log/ext_ioc_push.elg

  • For remote feeds (HTTP / HTTPS):
    • Web servers might return files with unauthorized or forbidden headers as a response. Make sure that file is fetched as expected and contains the correct information, whether it’s a STIX file or a CSV file, in: $FWDIR/external_ioc/feed_name_folder
    • For HTTPS remote feeds, if the certificate update process failed, you can skip the certificate verification. Run: export EXT_IOC_NO_SSL_VALIDATION=1 on the Security Gateway.
    • Check the Proxy configurations:
      • By default, the Security Gateway's proxy will be used
      • Override proxy configurations: --proxy <proxy:port>
      • Don't use proxy: --proxy <none>
This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment