Michael Leiter, Director, National Counterterrorism Center
John Michael McConnell, former Director of National Intelligence
*****
In war, you are what you do.
Michael Leiter championed the RAILHEAD PROJECT: he spent $500-million to move counterterrorism IT to a brand new, one-off/one-company proprietary database and search engine.
-- RAILHEAD failed to match Mr. Umar Farouk Abdulmutallab.
-- I replicated the reported RAILHEAD error, using the off-the-shelf linguistic search tool that National Archives uses to support research at the Presidential Libraries.
-- Scoring with Out-of-Africa info NEVER dropped Mr. Abdulmutallab off the the first page of candidates for the match.
-- Further, even a very simplified INDEX-SET pull got Mr. Abdulmutallab's tracking identifier from his Visa record. 100% identification. He would certainly have been transferred to the No Fly List.
Former-DNI McConnell screwed the pooch: clueless for quality control.
Testing RAILHEAD was McConnell's responsibility.
Contractor profit, foolishness, and no-balls quality control risked killing 289 people on Northwest Flight 253.
Sadly, bad-to-worse IT blunders BTF :::
The more we dig into this situation at ODNI and the National Counterterrorism Center, the worse it gets.
Yep, son, we have met the enemy, and he is us.
-- Walt Kelly
*******************************************
RAILHEAD blew $ 500-million for new, blank-page vaporware -- building in a design-time SNAFU blunder and covered up with bogus quality control.
The powers that be let the techies go on bribed_up/kicked_back/promised_lifetime_employment/Wednesday_night_poker_gamed/wife_getting_$125,00_
for_a_sales_gig.........
The usual Beltway Banditry. $500-million cannot be spent honestly and have minimal quality control.
-- Nobody writes a one-off full-service modern database system, much less a joke of a system with a string-search engine.
1.) You call Google.
2.) You call Oracle.
3.) You call somebody else in the linguistic search engine sector with 10,000 database programmers on staff and 100,000,000 end-users already kicking butt to make things work.
The modern technology makes linguistic searches and multiple-language linguistic searches. There are ISO standards: 2788 and 5964. That is for the driver logic -- the database systems chew up queries like kids going through Cheerios-and-bananas.
IBM once got $180,000,000 out of FBI for a dysfunctional rewrite of Documentum. That is IBM's tool for magazine publishers. Burning that money in the parking lot would have saved time. The Bureau's IT avoided quality control, but you knew that.
*********************************************
No Quality Control, No Quality.
What else ?
The reported error is a minor misspelling of the name. So let's replicate the core of the RAILHEAD identification system and let's give it the simplest likely identification information architecture -- a borrow-over from financial tracking software that supports compliance reports to IRS, Treasury, The Fed, DHS, and the like.
I also like the problem of supporting Arabic in Arabic. I don't know #### about Arabic. If its little kids and they talk slowly, its pretty to hear. Need help setting up the tests.
-- Design-Time Quality Control
Did a table group with the text fields in Standard Arabic. The head count is 1-billion people. Name, date of birth, place of birth, sex, nationality, height, weight, date of first entry, source of first entry, a unique synthetic identifier, and a status pointer -- a reference to a warehouse system that can contain an astonishing variety of different records.
INDEX-SET is jargon. All you need to know is that the linguistic query can be supported with information from ordinary database tables -- using indexes to get the info pulled before Sol burns out.
Data gets put in. Indexes are built for linguistic and relational access. A simple ETL input flow system gets built to enable Known Error data lines to be thrown in. A backup Data Guard and Flashback setup is installed -- vanilla there.
The simplest queries are run. This demonstrates that the overall system ties together. If you have George Washington in the system, you can find George Washington.
-- Proof Of Concept.
You build a system like my test system to demonstrate that the fundamental tasks -- here, finding candidate-terrorists -- are handled with the ordinary quality of inputs and recorded data.
You do need to clean up old data. You need to test auxiliary pieces -- do feeds from different identification sources use compatible Character Sets ? If not, how soon can that be corrected ? You worry with about 37 different design related groups of QC issues. You ask the questions.
The idea goes back to Air Force Systems Command. At least. AFSC kept planes in the air and didn't give away the farm on its main operations.
AFSC insisted on seeing things work in the small before they cut big checks.
I ask one question.
-- Can we match the SOB ?
Well, Oracle can match the SOB.
The instant, obvious, worst of it is that I can match Mr. Umar Farouk Abdulmutallab with a simple Scheduler routine without his name.
I can (and would) architect the program so that the system will run this match to Out-of-Africa data without human intervention -- driven solely by the request for processing coming in out of the Department of State.
Look at what we have coming out of Africa:
-- (a name string that we are discarding as unusable, possibly bogus)
-- Father's name and identifying document
-- Father's residence, date of birth, general descriptors
-- The indication that the individual had a prior U.S. Visa -- meaning at the least a Visa application went through
-- Date of Birth
-- Place of Birth
-- Dates related to earlier presence in a NATO country
You get the idea....
If RAILHEAD had the usual database programming capabilities -- running parallel subordinate searches -- there was no way to miss matching Mr. Abdulmutallab. Data in the main identification databases would be triggered with the family and descriptive data alone.
-- Running Degraded Linguistic Searches
I set up tests where the query data had Date of Birth and Place of Birth. This turned out to be too easy. I cut it back to Year and Country.
The underlying data comes from telephone records for names and synthetic descriptor fields.
The Oracle Text linguistic search engine eats this up.
The input is run with Standard Arabic at input from the keyboard, chosen to match the database Standard Arabic.
Its a no-brainer.
There is no way to force Mr. Abdulmutallab off the front page -- trying to move him down the list of candidates when we're trying to find a match to the Out-of-Africa Report. And then the system goes to its auxiliary pages (with manual intervention) and pulls up the Abdulmutallab-as-candidate data related to Mr. Abdulmutallab's father and the core data from his Visas to England and America.
I ran the query as "Umar Farouk" and he sat up as my # 3. That was with Name getting a 60% value for scoring and no routine to get tricky for missing surnames. The Status field on the screen blinks for the hits with Father and Visa.
*****************************************
Toss No Good Money After Bad
RAILHEAD is a failure.
We have never seen a commercial linguistic search engine -- not even the first, not-ready-for-beta Oracle Context "Cartridge" -- that was as bad as this RAILHEAD vapor-poo.
String search on the RAILHEAD pattern is not linguistic search.
And even for that one task, as though NCTC could live with string search, Scoring is everything.
Databases have to be able to combine the linguistic searches with ordinary relational searches. All the ordinary data is in relational tables.
(If you want to scream, "DATA WAREHOUSE !" then you know too much for your own good. Have a deep toke. Put aside mindless fault-finding. Relax.)
As noted, finding Mr. Abdulmutallab without his name went fine. Swimmingly. He popped right out. The Out-of-Africa descriptive data is overwhelming.
Someone WAS taking notes......
Possibly: no one in any position of authority at either the Office of the Director of National Intelligence or at the National Counterterrorism Center has had any skill, whatsoever, at implementing database systems. Ever.
I have not placed blame to Dennis C. Blair, who became Director of National Intelligence on January 29, 2009. That infernal RAILHEAD Project had been run to ruin before his arrival. The fakery of bogus tests and lies had been constructed. Souls had been purchased. Second-career double-dippery was rampant.
But if Dennis C. Blair leaves RAILHEAD in place -- in any form whatsoever -- then Dennis C. Blair will deserve to be drawn-and-quartered as per the Common Law response to High Treason. With the full traditional insults from the day of Elizabeth I.
There is no excuse.
Cowardice from a former Admiral, certainly not.
********************************************************
We can talk briefly about this:
The Terrorist Threat to Inbound U.S. Passenger Flights: Inadequate Government Response
Homeland Security Affairs
The Journal of the Naval Postgraduate School Center for Homeland Defense and Security
Published UNCLASSIFIED to open internet: January 2009. Left up and mirrored.
Sets up the Big Opportunity. Disposition of explosive-detection equipment by TSA -- specifically not overseas and info that other countries were not picking up the slack. Similar to FAA not getting the airliner cockpits locked and issuing "surrender" crew instructions.
1.) "Inadequate Government Response" paper. 2.) RAILHEAD dysfunction.
It wasn't Mr. Abdulmutallab's day to die.
Any competent counterterrorism system would have resources committed to tracking Top Secret information flows out of critical personnel -- the like of TSA senior consultants. "Come get us HERE !!" was not the theme of the day.