HTML Scraping to Migrate Web Content

[Stephen Bell]

We're in the process of moving from static HTML pages (Dreamweaver templates) to a content management system for our main university web site.  A lot of CMSes have import utilities that can use CSV (Comma Separated Value) files or XML.

I am looking for a way to grab all of the data on our old pages and put them into one giant CSV file (or at least a folder at a time) for bulk import into our new site.  Finding the content areas on our pages would be pretty easy, since they're already clearly marked in the code (ie. "<!-- InstanceBeginEditable name="Text" -->").

Does anyone have experience doing something like this?  Something like Mozenda (http://www.mozenda.com/) looks promising, but I'd prefer something that is a little cheaper or at least doesn't charge by the page.

[Jason Garber]

I'm nearing the end of a 5-month CMS migration from Dreamweaver. I did it by hand (read: work-study students) because the HTML in Dreamweaver had 10 years of cruft built up and I didn't want that in my shiny new CMS. I could have done it in a couple days, though, with a Ruby script. If you don't have someone with those skills, I'm sure you could hire someone (or yours truly) for just a few hours to do it.

Jason

[Steve]

I'd second the notion that depending on how recently those pages were created, you could be bringing along font tags and the associated "cruft."

I'm working on something similar, parsing through HTML and producing an XML markup used by our new CMS. I just want to get stuff in. My tool of choice is PERL given all the pattern matching abilities of that language.

[Shelley Keith]

Stephanie Leary built a really great plugin for WordPress that does just this. You could pull your static content into a WP install and export to XML. Might be a quick and easy way to accomplish the task.

[Pemmymac]

Hi,

Just curious, what kind of CMS will you be using?

Pem

[Stephen Bell]

dotCMS is the frontrunner at the moment.

[Chris Wiegman]

I've helped with a few such migrations.Getting the data areas themselves aren't hard. A good developer could write a script for it rather easily. It's the formatting, accessibility, etc that is a problem especially with DW where there tends to be a lot of Garbage code. I've tried plugins and other methods, but in the end your best bet might be something on the lines of a student worker and a text editor. Anything else can cause all sorts of extra problems down the line.

[Ben Lobaugh]

Stephen,

How deeply have you delved into this? Have you thought about site structure? Is that going to change at all? What about any attachments like PDFs and images that are part of the content?

The Typo3 extension we built not only imports the HTML files into the db, it also imports any files, such as PDF (you specify file types), rebuilds the links to work with the CMS, and if you have Dreamweaver template tags around your content it can target only that area.

Of course there is still some manual editing we are doing, but I processed at 35k+ file site in less than an hour and have only needed to make relatively small tweaks on what was imported (not counting actual restructuring)

The extension is still a little rough, but fully workable. We were planning on releasing it back to the Typo3 community after a little cleanup, but if you would like to take a gander at it please feel free to talk to Zac or myself

[Kerri Hicks]

I have a BBEdit Text Factory that does this. If you're on a Mac, you can download BBEdit from barebones.com (free for 30 days), go to the menu that looks like a gear, open the Text Factories folder, and then put in the file below (once you've extracted it from the .zip archive). Once you put it into the folder, you can run the Factory over a whole folder/site full of documents.

http://www.uri.edu/news/hicks/uwebd_clean_DW.textfactory.zip

[Chad Hietala]

I would look into YQL (Yahoo Query Library). You can build your own query that delivers either an XML tree or JSON structure then parse that query with any language. I've never used in the matter you are looking to do but it's worth a shot.

[Andy Fore]

Well, it doesn't look like it will do the job en masse but DW CS5 can export the template data as an XML document. You can export the XML two ways, one where the editable sections names as the XML tags and one that uses DW XML tags.

[Cory Watson]

I recently built a migration script as a proof of concept that my college could make the change to Drupal pretty seamlessly.

 

Our current CMS (reddot) has the ability to output valid XHTML files. I built a script in classic ASP script (am currently porting it to PHP) that scraped these XHTML files using Xpath and output them into a CSV file. I just made sure to consistently name elements (e.g. - <div id="header">) so that Xpath could find them.