Newsgroups: alt.religion.pagan.evil,alt.usenet.legends.lester-mosley
Sent: Thursday, September 25, 2008 10:41 PM
Subject: flaws and U

> http://www.computerworld.com/action/article.do?command=printArticleBa...
> When the meteor and the 1PB database collide
> Craters? No, ginormous amounts of celestial information in need of storage
> Eric Lai
> August 8, 2008 (Computerworld) Our fascination with the prospect of
> asteroids smashing into the Earth is as deep as the craters that can
> result
> from such cosmic fireballs. Think of all the movies Hollywood has made,
> from
> little-seen B flicks such as A Fire in the Sky to campy cult classics such
> as Night of the Comet to scientifically shaky blockbusters such as Meteor
> and Armageddon.

> The 1990s was also awash with news of rocky passersby such as Comet
> Hale-Bopp and Comet Shoemaker-Levy 9, which unleashed fragments up to two
> kilometers wide upon Jupiter in 1994.

> Once dismissed as the province of fringe cult groups, the fear of what
> astronomers call "impact events" turns out, thanks to improved satellite
> and
> telescopic monitoring, to be not so irrational after all.

> Pan-STARRS on patrol

> The latest and most ambitious to detect 'near-Earth objects' (NEO) is the
> Panoramic Survey Telescope and Rapid Response System, or Pan-STARRS.

> A joint venture of the University of Hawaii, a number of other schools and
> the U.S. Air Force, Pan-STARRS is today testing a telescope mounted with
> the
> finest digital camera in existence, which boasts a resolution of 1.4
> billion
> pixels.

> When Pan-STARRS is fully operational several years from now, it will have
> four telescopes, each with a 1.4-gigapixel camera.

> That will give Pan-STARRS a wider, faster and more-powerful view into
> space,
> and will enable it to meet its mandate of tracking virtually all NEOs
> larger
> than 300 meters in diameter as well as many smaller NEOs.

> It will have plenty to see. About once a year, an asteroid of five to 10
> meters in diameter explodes in the Earth's upper atmosphere, releasing as
> much energy as the atomic bomb used at Hiroshima. And if one slips
> through,
> it can cause a lot of damage -- even if it's not a big one.

> The asteroid behind 1908's Tunguska Event was only about 50 meters in
> diameter, but it created an explosion equivalent to 10 to 15 megatons of
> TNT
> (about 1,000 times the Hiroshima bomb), knocking over an estimated 80
> million trees in Siberia and causing an earthquake that's estimated to
> have
> measured a 5.0 on the Richter scale (which had not yet been invented at
> that
> time). And we are due for another even like that within 200 years,
> according
> to the late astronomer Eugene Shoemaker.

> With just a single telescope, Pan-STARRS already generates 1.4 terabytes
> of
> raw image data nightly. Compressing, storing and crunching that data in an
> economical fashion turns out to be a feat of database engineering as
> impressive as the collection process.

> Rather than turning to an expensive supercomputer equipped with hundreds
> or
> thousands of processors, Pan-STARRS will use a cluster of 50 PC servers
> connected to 1.1 petabytes of disk storage via fast Infiniband networking
> gear, according to Alex Szalay, a physics and astronomy professor at Johns
> Hopkins University and one of the architects of Pan-STARRS' database.

> And rather than using a database management program better-known for
> ultralarge data warehouses, such as IBM's DB2, a TeraData system or Oracle
> Database, Pan-STARRS will use Microsoft Corp.'s just-released SQL Server
> 2008.

> Weighing the benefits

> Even Microsoft would probably admit that despite improved data compression
> and a resource governor to manage multiple workloads, SQL Server 2008 is
> not
> the most intuitive choice for this clustered, "scaled-out" schema.

> "SQL Server 2008 takes us to the next level, but that is within the
> 'scale-up' model," said Ted Kummert, a vice president in Microsoft's data
> and storage platform division, this week during a conference about the
> launch of the upgraded database. Rather, Microsoft's recent acquisition of
> DATAllegro Inc., a start-up vendor focused on large data warehouses, "will
> take us to the greatest level of scale-out," he said.

> There are several reasons, though, why Pan-STARRS went with SQL Server
> 2008.

> One is cost. Deploying Pan-STARRS will cost just $750,000, thanks to the
> low
> cost of the PC hardware and the heavy academic discounts offered by
> Microsoft for SQL Server and Windows Server 2008.

> "People in academia are always operating on a shoestring budget, so we
> wanted to be able to create something others could emulate," Szalay said.

> More important, however, is Microsoft's long involvement with the
> astronomical community, especially via its technical ambassador, Jim Gray.
> The noted database researcher, who disappeared at sea in early 2007 and is
> now presumed dead, was instrumental in building earlier databases, such as
> TerraServer, a massive free Web archive of satellite pictures of the Earth
> stored in SQL Server, and the 40TB SkyServer, a similar repository of
> astronomical images.

> Indeed, the distributed database platform that Pan-STARRS (and, it is
> hoped,
> other applications) will run on is called GrayWulf in Gray's honor.

> "Gray worked with us for more than a decade. All the credit should go to
> him," Szalay said.

> "He changed astronomy as we know it," said Maria A. Nieto-Santisteban, a
> software engineer at Johns Hopkins and the technical lead of the
> Pan-STARRS
> project. "We still ask ourselves, 'How would Jim do this?'"

> From magnifying glasses to megastorage

> Astronomers first began storing data digitally in the mid-1970s, shortly
> after they began replacing conventional photographic plates with digital
> camera technology.

> Efficiency-wise, digital cameras were still a vast improvement over those
> photographic plates, which required astronomers to hunch over them with
> magnifying glasses, counting galaxies and stars. But the digital image
> resolution back then left something to be desired -- just 260,000 pixels.

> Data storage was also crude. Image data was and is still stored in a
> low-level format based on 80 character-long punch cards. But the flat
> files
> used to store the data proved difficult to search and otherwise
> manipulate.

> Gray guided the building of SkyServer, which holds 100 billion rows of
> data
> and 1 million distinct IP addresses, and serves 10,000 to 15,000
> professional astronomers as well as countless schoolchildren who use
> SkyServer to complete astronomy reports.

> Pan-STARRS, which Gray helped conceive, will be far larger, containing, by
> the end of 2010, 300TB of data, with some individual tables as large as
> 20TB, Szalay said. The repository will include data on more than 140
> billion
> cosmic objects and 5.5 billion actively tracked ones.

> Though Pan-STARRS won't use up all 1PB of storage for many years, it will
> still rank as one of the world's largest databases.

> Since Pan-STARRS is set up as a clustered system, the data will be
> partitioned, with a separate names database serving as the index. Since
> most
> cosmic objects don't have names such as Earth or Alpha Centauri, most
> searches will be done via a graphical interface that, according to Szalay,
> "looks and feels a lot like MapQuest or Google Maps."

> Besides being used to look up data on individual stars or galaxies,
> Pan-STARRS will also be used to do some deep data mining -- astronomical
> intelligence, if you will. For instance, Szalay hopes to import old
> astronomical data from the pre-digital age and run the information through
> a
> spatial cross-matching engine in order to create a master database that
> links all past and present data about every single star or planet.

> Pan-STARRS will also serve as a cloud database for outside astronomers,
> who
> will be allowed to remotely run queries and store results within
> Pan-STARRS.
> An initial difficulty, Nieto-Santisteban acknowledged, is that most
> astronomers are used to writing applications in C++, not SQL.