Author Topic: Downloading all your posts from The Old Place.  (Read 14096 times)

Wowbagger

  • Former Sylph
    • Stuff mostly about weather
Downloading all your posts from The Old Place.
« on: 27 March, 2008, 11:49:09 pm »
My son downloaded all my posts for me using this simple php script:-

<?php
for ($i=0;$i<702;$i++) {
  system("wget --html-extension --load-cookies ~/cookies.txt http://www.anothercyclingforum.com/index.php?action=profile\;u=1230\;sa=showPosts\;start=".($i*15));
}
?>


For this to work you need php installed and to log into the ACF forum in your web browser then locate your cookies.txt file or export cookies if using IE. Replace the "~/cookies.txt" with your location for the file. The 702 in the "for" statement was my number of pages of posts. You need to find your own number of pages using your profile and viewing your posts.

The pages will be downloaded into the current directory.

I hope this all makes sense. It didn't mean a thing to me.
The loss of humanity I could live with.

Re: Downloading all your posts from The Old Place.
« Reply #1 on: 27 March, 2008, 11:52:26 pm »
I've done something similar. I'll work on some magic to convert the resulting HTML files into something more readable in the future.
"Yes please" said Squirrel "biscuits are our favourite things."

Re: Downloading all your posts from The Old Place.
« Reply #2 on: 27 March, 2008, 11:53:17 pm »
Beyond me, I fear.

Maladict

Re: Downloading all your posts from The Old Place.
« Reply #3 on: 27 March, 2008, 11:53:52 pm »
My son downloaded all my posts for me using this simple php script:-

<?php
for ($i=0;$i<702;$i++) {
  system("wget --html-extension --load-cookies ~/cookies.txt http://www.anothercyclingforum.com/index.php?action=profile\;u=1230\;sa=showPosts\;start=".($i*15));
}
?>


For this to work you need php installed and to log into the ACF forum in your web browser then locate your cookies.txt file or export cookies if using IE. Replace the "~/cookies.txt" with your location for the file. The 702 in the "for" statement was my number of pages of posts. You need to find your own number of pages using your profile and viewing your posts.

The pages will be downloaded into the current directory.

I hope this all makes sense. It didn't mean a thing to me.

Yep that is about what I was thinking of.

You also need to replace the u=1230 with whatever your own ACF user ID is.

I suggest it might also be wise to have a short delay between each
download in the loop (e.g. sleep(5)).  That would avoid hammering
the server, especially if a lot of people try the same trick.

Maladict

Re: Downloading all your posts from The Old Place.
« Reply #4 on: 27 March, 2008, 11:54:22 pm »
Beyond me, I fear.

What OS do you use?

Re: Downloading all your posts from The Old Place.
« Reply #5 on: 28 March, 2008, 12:00:53 am »
Winders XP.

rae

Re: Downloading all your posts from The Old Place.
« Reply #6 on: 28 March, 2008, 12:21:11 am »
Wouldn't it be better to pull entire threads?

Maladict

Re: Downloading all your posts from The Old Place.
« Reply #7 on: 28 March, 2008, 02:24:45 am »
Wouldn't it be better to pull entire threads?

Not sure.  I was first of all interested in saving all my own work.

Maladict

Re: Downloading all your posts from The Old Place.
« Reply #8 on: 28 March, 2008, 02:28:39 am »
My version:

Code: [Select]
#!/usr/local/bin/python

import os

f = open("/tmp/acf-saves-list","w")

for i in range(120):
    f.write("http://www.anothercyclingforum.com/index.php?action=profile;u=1465;sa=showPosts;start=%d\n"%(i*15))

f.close()

os.system("wget --wait=10 --random-wait --html-extension -p --convert-links --load-cookies ~/cookies.txt -i /tmp/acf-saves-list")


This has the (small) advantage of downloading all the prerequisites for the
pages so it looks correct even when ACF goes tits-up.

You may also want to save all your PMs.

Re: Downloading all your posts from The Old Place.
« Reply #9 on: 28 March, 2008, 03:06:28 am »
I used wget, and I think I have all my posts.    I'll trawl tomorrow and save "interesting" threads too.

I'm more interested in getting these .html web pages on my desktop converted into something useable though - at the moment they're identical to the on-line version.

PaulF

  • "World's Scariest Barman"
  • It's only impossible if you stop to think about it
Re: Downloading all your posts from The Old Place.
« Reply #10 on: 28 March, 2008, 08:38:44 am »
Please can we have an 'Idiot's Guide' to this?

I don't know how to run a php script :( To be honest I wouldn't know one if it hit me in the face :D

clarion

  • Tyke
Re: Downloading all your posts from The Old Place.
« Reply #11 on: 28 March, 2008, 09:01:05 am »
That's all clever magic.  I'm not sure I can get enough steam up on this machine to breach the ICBA and incompetence thresholds....
Getting there...

Mr Larrington

  • A bit ov a lyv wyr by slof standirds
  • Custard Wallah
    • Mr Larrington's Automatic Diary
Re: Downloading all your posts from The Old Place.
« Reply #12 on: 28 March, 2008, 10:52:36 am »
Please can we have an 'Idiot's Guide' to this?

I don't know how to run a php script :( To be honest I wouldn't know one if it hit me in the face :D

Seconded.  Can it be done with FORTRAN ???

(Actually we don't even have a FORTRAN compiler here chiz)
External Transparent Wall Inspection Operative & Mayor of Mortagne-au-Perche
Satisfying the Bloodlust of the Masses in Peacetime

PaulF

  • "World's Scariest Barman"
  • It's only impossible if you stop to think about it
Re: Downloading all your posts from The Old Place.
« Reply #13 on: 28 March, 2008, 11:17:00 am »
Please can we have an 'Idiot's Guide' to this?

I don't know how to run a php script :( To be honest I wouldn't know one if it hit me in the face :D

Seconded.  Can it be done with FORTRAN ???

(Actually we don't even have a FORTRAN compiler here chiz)

Shouldn't be a problem, a good programmer can write FORTRAN in any language :)

Re: Downloading all your posts from The Old Place.
« Reply #14 on: 28 March, 2008, 11:46:43 am »
Sometimes I feel distinctly dumb compared to people who can do this sort of thing!

Mr Larrington

  • A bit ov a lyv wyr by slof standirds
  • Custard Wallah
    • Mr Larrington's Automatic Diary
Re: Downloading all your posts from The Old Place.
« Reply #15 on: 28 March, 2008, 01:21:08 pm »
Please can we have an 'Idiot's Guide' to this?

I don't know how to run a php script :( To be honest I wouldn't know one if it hit me in the face :D

Seconded.  Can it be done with FORTRAN ???

(Actually we don't even have a FORTRAN compiler here chiz)

Shouldn't be a problem, a good programmer can write FORTRAN in any language :)

Crses!  Pwned agn :(
External Transparent Wall Inspection Operative & Mayor of Mortagne-au-Perche
Satisfying the Bloodlust of the Masses in Peacetime

Re: Downloading all your posts from The Old Place.
« Reply #16 on: 28 March, 2008, 02:22:18 pm »
It's worth bearing in mind, that the more that people do this sort of thing, the faster it is likely to bring acf down, since eventually it'll reach some sort of bandwidth limit, and I doubt sam will rush to deal with it.

If there are certain things which you must have, then all well and good, but I wouldn't just download every post you ever made, "just in case", since out of context of the threads they are in, they are likely to look a bit strange anyway.
Actually, it is rocket science.
 

Woofage

  • Tofu-eating Wokerati
  • Ain't no hooves on my bike.
Re: Downloading all your posts from The Old Place.
« Reply #17 on: 28 March, 2008, 02:27:41 pm »
It's worth bearing in mind, that the more that people do this sort of thing, the faster it is likely to bring acf down, since eventually it'll reach some sort of bandwidth limit, and I doubt sam will rush to deal with it.


What I was going to add. Well said!
Pen Pusher

Re: Downloading all your posts from The Old Place.
« Reply #18 on: 28 March, 2008, 02:41:47 pm »
I just checked my browser history. On one particular day I looked at 1200 unique URLs at ACF.

Assuming each page has, on average, 10 posts that means I used up approximately the same bandwidth that day as I would have downloading 12000 posts, more than double the number I actually did.

The total file size for my 5500 posts was "only" 15MB.
"Yes please" said Squirrel "biscuits are our favourite things."

Nick H.

Re: Downloading all your posts from The Old Place.
« Reply #19 on: 28 March, 2008, 04:47:11 pm »
WHY OH WHY OH WHY can't we import all the sodding posts into YACF?  ??? Nothing else makes sense. This is all turning into a giant pain in the arras.  >:(

Re: Downloading all your posts from The Old Place.
« Reply #20 on: 28 March, 2008, 05:26:37 pm »
I just checked my browser history. On one particular day I looked at 1200 unique URLs at ACF.


"working from home" again, eh?

 ::)

Re: Downloading all your posts from The Old Place.
« Reply #21 on: 28 March, 2008, 05:27:39 pm »
Beats "tele-working", i.e. lots of tele and not much work.
"Yes please" said Squirrel "biscuits are our favourite things."

Mrs Pingu

  • Who ate all the pies? Me
    • Twitter
Re: Downloading all your posts from The Old Place.
« Reply #22 on: 28 March, 2008, 05:50:39 pm »
Is someone copying a load just now? Because old ACF is not working atm in The Igloo.
Do not clench. It only makes it worse.

Re: Downloading all your posts from The Old Place.
« Reply #23 on: 30 March, 2008, 03:06:17 pm »
WHY OH WHY OH WHY can't we import all the sodding posts into YACF?  ??? Nothing else makes sense. This is all turning into a giant pain in the arras.  >:(

That is entirely possible, but unless somebody has the right credentials to be able to dump the entire database into one massive sql file, it's unlikely to happen.

If nobody has that access, I doubt the-guy-who-ran-ACF would be particularly forthcoming with the required details.

And my hacking days are long gone  :P
Those wonderful norks are never far from my thoughts, oh yeah!

Basil

  • Um....err......oh bugger!
  • Help me!
Re: Downloading all your posts from The Old Place.
« Reply #24 on: 30 March, 2008, 03:23:23 pm »
Fortunately, I no interest at all in my posts.  It's you lot I in back to re-read.  So this isn't much help to me.
Admission.  I'm actually not that fussed about cake.