Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

OSINT in 2013

ISec Open Forum - August 2013
by

Jonathan Cran

on 13 November 2015

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of OSINT in 2013

ISec Partners Open Forum
August 29, 2013
Who is this presentation for?
OSINT is
kid stuff, right?
Entity extraction refers to the process of extracting structured information from unstructured data sources.

Such structured data may include:

Entities, events and documents
Properties associated with entities, events and documents
Relationships between entities, events and documents
In Palantir, entity extracted documents are represented in the DocXML format.
OSINT
Entity Extraction
Data Gathering
Scraping
Discovery
Distillation
Delivery
Intelligence
Open Source
Finding Data Sources
Open-source intelligence (OSINT) is a form of intelligence
collection management t
hat involves
finding,

selecting
and
acquiring
information from
publicly available sources
and
analyzing
it to produce
actionable
intelligence.
Data Collection
Recon
Passive
Google Dorking
Semi-Passive
DNS
Database
Google
Acquiring Data
Creating Information
Analyzing Information
References

Forrester
BasisTech
Knight+
Palantir
BrightPlanet
TapirRecon http://www.github.com/pentestify/tapir
Information brokers
Traditional Sources
Axciom - http://www.databyacxiom.com/
LexisNexis - http://www.lexisnexis.com/
ChoicePoint (now LexisNexis) - http://www.lexisnexis.com/
KnowX (now LexisNexis) - https://www.knowx.com/
DOCUSEARCH - http://www.docusearch.com/
Discreet Data - http://www.discreetdata.com/
MasterFiles - http://www.masterfiles.com/
InfoChimps - http://www.infochimps
Intelius - http://www.intelius.com
LEXIS NEXIS
To perpetrate the scam, the fraudsters would set up fake mail boxes and then use information obtained on LexisNexis to open credit cards in the victims' names. The criminals were able to obtain names, dates of birth, and even Social Security numbers from the data broker.
OSS Data Sets
Normalize
Logfiles
Internal
Databases
Data model is key
Connections are important
Connectedness
Relatedness
Structure
Kapow
<MaltegoMessage>
<MaltegoTransformResponseMessage>
<Entities>
<Entity Type=’Phrase’><Value>Hello $entityValue</Value></Entity>
</Entities>
</MaltegoTransformResponseMessage>
</MaltegoMessage>
Maltego
Acxiom
"Our policy clearly states that we 'provide information products which include financial information, Social Security number and other related information where permitted by law,' and that this information is 'provided to government agencies for the purposes of verifying information, employment screening and assisting law enforcement.'"
In 2003, over 1.6 billion customer records were stolen during the transmission of information to and from Acxiom's clients; the information included names, addresses, and email addresses. Acxiom's firewall was not breached, and no databases were accessed by the hackers.[27] Prosecutors described the 2006 case against the hacker accused of stealing the data as the "largest ever invasion and theft of personal data" ever tried
Get rid of bogus data
Filter out interesting data
Depends on how you view the world
Palantir Government
Paterva
Java client app, can run local / remote
Great data analysis capabilities
Good integration / API / (TAS)
Casefile
Can pull from database (SQLTAS)
>67,000 users
Automation solution
Scraping
Create a "Live search" - basic web service which performs scraping on the backend
Conditional Logic
Interacts with databases
Outputs to database, xml, csv, web services, etc
Automate browsers
Integrates nicely with analysis tools
TAPIR
Made by Palantir
Deployed at Fortune 50 companies
Product is a child of PayPal
Incredible analysis platform
Logfiles, internal databases, OSINT sources
Data
Traditional pentesting tools just find data
Data isn't the flippin problem
We're swimming (sinking) in data
Open and Closed sources ("Deep web")
Many OSINT source collections
http://rr.reuser.biz/
http://www.uk-osint.net/favorites.html
https://sites.google.com/site/greynetwork2/home/osint-resources
http://www.yougetsignal.com/
etc
http://law.lexisnexis.com/infopro/zimmermans/disp.aspx?z=1752
http://www.fso-online.com/home_login.cfm?sid=49170561
http://www.social-engineer.org/framework/Social_Engineers:_Information_Brokers
Domain
DNS Name
MX record
NS record
IP Address
Netblock or network
AS number
Web site
URL
Phrase
Document
Person
Email address
Affiliations
Location
Phone number
http://www.github.com/pentestify/tapir
Entities
Module Based Tasks
Parent / Child relationships
Ruby on Rails
BSD!
Microstrategy
Trilogy Software
Mapview
Arcview
Omniture's Discover On Premise
RecordedFuture
Basis Technology
Kapow
Encase
Analyst Notebook
KnightX+
Cogito


http://blog.palantirtech.com/2009/11/06/palantir-like-an-operating-system-for-data-analysis/
A hierarchical type system of the real-world objects that human experts use to think about this problem. We call these PTObjects, short for “Palantir Objects”.

A type system of properties that will contain the data describing these PTObjects. PTObjects are essentially typed containers for properties. This is where most of the detail of the ontology lies.

A type system of possible relationships between different types of PTObjects.

The data sources are mapped into the ontology

The data are composed into real-world objects.

The server exposes Palantir “system calls”

Most of the time and effort in machine learning is spent getting the data into a form that you can actually apply an algorithm to!

It’s about the start of the analysis age
http://www.charlierose.com/view/interview/10549
If we as a country understand what the danger is, then it's possible to fight this without giving up our civil liberties

Democracies tend to win when the people get behind it

What can we as citizens do (We as Americans) that allows us to experience the liberty we want, and stop them?

Educate the public about the threat

If we don't believe in what we're doing we can't win

Software has democratized espionage
Infochimps
LexisNexis Group provides computer-assisted legal research services.[2][3] During the 1970s, LexisNexis pioneered the electronic accessibility of legal and journalistic documents.[4] In 2006, the company had the world's largest electronic database for legal and public-records related information.[5]
In 2000, LexisNexis purchased RiskWise, a St. Cloud, Minnesota company.[12] In 2002 it acquired a Canadian research database company, Quicklaw. In 2004, Reed Elsevier Group, parent company of LexisNexis, purchased Seisint, Inc, of Boca Raton, Florida.[13] Seisint housed and operated Multistate Anti-Terrorism Information Exchange (MATRIX).
According to a company news release, LexisNexis hosts over 30 terabytes of content on its 11 mainframes (supported by over 300 midrange UNIX servers and nearly 1,000 Windows NT servers) at its main datacenter in Miamisburg, Ohio.[17]
http://www.infochimps.com/search?view=list&price_category=&has_categories=&dataset_type=&order=balanced&tags=&query=ip+address
Google
DNS / Whois / Robtex
Hoovers
Web Scraping
EDGAR
IP Geolocation
Email addresses
Social Networks
Pastebin
PublicData.com, et al
OSS Databases
State Websites
Job Sites
National property — Acxiom offers
the most comprehensive national
property database available in the
market today. Updated monthly,
national property includes 1,575
county assessor and 700 sales
records, making it the database of
choice for all real-property needs.

Comprehensive report — Acxiom
provides a detailed view of
information for a target individual.
The individual’s information
encompasses multiple sets of
data, which include the following:
the results from the Find People,
address history, alias, phone
history, relatives, associates (people
and corporations), concealed
weapons, licenses (drivers, vehicle,
hunting and fishing, professional
and pilot), property information,
and voter information
Acxiom’s combined information produces
some of the best hit and contact rates in
the industry, enabling you to consistently
recognize the consumer requested search
information even with name changes,
unreported moves or missing data

Acxiom leads the industry in fraud detection and identity verification of individuals with little to no credit history, enabling you to locate, recognize and verify individuals others cannot
Others
Not for commercial use!
Maximum of 12 results per transform
You need to register on our website to use the client
API keys expire every couple of days
Runs on a (slower) server that is shared with all community users
Communication between client as server is not encrypted
Not updated until the next major version (and we know there are some bugs)
No end user support – you are on your own..
No updates of transforms on server side
Entity -> Transform -> Entities
DEMO
built_with.rb
convert_netblock_to_hosts.rb
convert_netsvc_to_webapp.rb
dns_common_guess.rb
dns_forward_lookup.rb
dns_reverse_lookup.rb
dns_srv_brute.rb
dns_sub_brute.rb
dns_tld.rb
dns_tld_brute.rb
dns_txt_lookup.rb
dns_zone_transfer.rb
edgar_detail.rb
edgar_search.rb
example.rb
fuzz_port.rb
geolocate_host.rb
google_search.rb
hoovers_detail.rb
hoovers_search.rb
import_leadlander_list.rb
import_shodan_xml.rb
linkedin_api_query.rb
nmap_scan.rb
nmap_scan_web.rb
probe_port.rb
robots_txt.rb
social_profile_search.rb
spider_web_application.rb
twitpic_photo_locations.rb
twitter_api_query.rb
usernames_guess.rb
whois.rb
account.rb
base.rb
dns_record.rb
doc_file.rb
email_address.rb
facebook_account.rb
finding.rb
host.rb
image.rb
klout_account.rb
linkedin_account.rb
net_block.rb
net_svc.rb
organization.rb
parsable_file.rb
parsable_text.rb
pdf_file.rb
person.rb
physical_location.rb
search_result.rb
search_string.rb
twitter_account.rb
username.rb
web_application.rb
web_form.rb
web_page.rb
xls_file.rb
Data Location
Data Selection / Aquisition
Data Filtering / Analysis
Data Presentation
Separate Components
Gathering Tools
nslookup / dig / whois
Fierce
SEAT/Goolag
theHarvester
Metagoofil
ServerSniff
Hoovers
DomainTools
CentralOps
Robtex
Pipl
Wigle.net
Spokeo
Namechk
FOCA
Cree.py
etc etc etc
Analysis Tools
Netglub
Tapir
Maltego


Planning and Direction
Finding Data Sources
Acquiring Data
Transform Data -> Information
Analyzing Information
Transform Information -> Intelligence
Intel Dissemination

Defining OSINT
OSINT in 2013
ZOMG SLIDE REMOVED
Google
Baidu
Bing
Yahoo
Radio Transcripts
forums
usenet
Facebook
Orkut
LinkedIn
Google Plus
Meetup
Tumblr
Twitter
County DBs
State DBs
EDGAR
Corpwatch
http://www.defense.gov/contracts/
LexisNexis
Public Records - brbpub.com/
Newspapers
Magazines
pastebin
Google Maps
Google Earth
Bing Maps
Government Contracts
Infochimps
Axciom

Glassdoor
Scanning
Open APIs
Commercial APIs
Purchase Data Sets
Scraping
Find and maintain relationships
OSINT In the context of Intelligence Gathering
Knowing more about prospects
Knowing more about your customers
Knowing more about your competitors
Knowing more about your business threats
Business Threats
Finding Breaches
- Leaked credentials
- Leaked business information
- Rogue employees
Finding Malware
Finding upcoming malicious activity
Know what your attackers know

R
Maltego
Palantir
Write your own scripts
https://www.privacyrights.org/online-information-brokers-list
http://www.social-engineer.org/framework/Social_Engineers:_Information_Brokers
Right now in Conway, Ark., north of Little Rock, more than 23,000 computer servers are collecting, collating and analyzing consumer data for a company that, unlike Silicon Valley’s marquee names, rarely makes headlines. It’s called the Acxiom Corporation, and it’s the quiet giant of a multibillion-dollar industry known as database marketing.
In investor presentations and interviews, Acxiom executives have said that the company — the subject of a Sunday Business article last month — has information on about 500 million active consumers worldwide, with about 1,500 data points per person. Acxiom also promotes a program for consumers who wish to see the information the company has on them.
Several days later, Ms. Barrett Glasgow called to explain the delay in processing: Acxiom receives, on average, fewer than 100 requests a year from consumers, she said, and my check had “ended up on someone’s desk that was on vacation.” She said she would look into why company representatives hadn’t returned my voice mail message.
http://www.nytimes.com/2012/07/22/business/acxiom-consumer-data-often-unavailable-to-consumers.html?pagewanted=all
DNS
Fierce.pl
WHOIS
SHODAN
Passive Recon
Netcraft
Maltego
FOCA

Entity Extraction
Filtering
Analysis
reverse phone search
Jigsaw
Hoovers
EDGAR
IRC
Yahoo Local
El Futuro
"Big Data"
Graph Databases
Stream Processing
http://techcrunch.com/2012/10/27/big-data-right-now-five-trendy-open-source-technologies/
Drill
Dremel (Bigquery)
R
Gremlin
Giraph
SAP Hana
D3
Analysis Platforms
http://www.indeed.com/q-Osint-Analyst-jobs.html
attach_note.rb
goofile_doc.rb
goofile_pdf.rb
goofile_xls.rb
google_dork_search.rb
google_file_download.rb
google_hostname_search.rb
google_subdomain_search.rb
traceroute.rb
web_screenshot.rb
zmap_identify_others.rb
zmap_scan.rb
Questions/Comments?

Takeaways
Know what your attacker knows about you
Have an attacker mindset for your organization
Business Intelligence can use OSINT
OSINT is more than gathering flippin data manually
Data brokers are scary, need more control
Big Data presents new opportunities, problems
Check out Tapir
@jcran
Full transcript