Website traffic and server logs

[nikki]

This is going to be necessarily scant on background information, but can anyone think of any circumstances under which traffic to a website might not show up in the server logs.

(WordPress installation with a privacy plug-in so you have to be logged-in to view the content, other than that you get a WordPress log-in screen. Not accessing the server logs directly, but using Webalizer to view.)

ETA: I should have said traffic from particular individuals not showing up, whilst traffic from others is.

[Afasoas]

It's unlikely you'd be in a scenario where some and not all traffic is logged.

But it's impossible to say without knowing what web server is use (Apache?) and how the logging is configured. I'm guessing the website is hosted with an ISP on a server managed through some sort of control panel?

[Pickled Onion]

Agreed, any production web server would log every request.

There are only two^* sane reasons I can think of:

1) Webaliser is failing to show you all the lines in the log
2) This traffic is not actually hitting the webserver - for example they are getting the pages from a caching proxy

Personally I'd go with (2) but as afasoas says, more details are needed.

* - another possibility is this is a load-balanced server and you're only seeing logs from one of the servers, but this is unlikely.

[Jaded]

Are you looking at raw log files or some kind of Stats analysis. It is possible to filter IP addresses from log file reports, so this may be happening? Or the people are accessing via a TOR or proxy?

[nikki]

Thanks all.

It's unlikely you'd be in a scenario where some and not all traffic is logged.

*nods* This is my understanding of the situation. Once I get to "highly unlikely" I can kick some ass.

But it's impossible to say without knowing what web server is use (Apache?) and how the logging is configured. I'm guessing the website is hosted with an ISP on a server managed through some sort of control panel?

I could probably get web server and logging config from the host's support service.
Yes to the control panel. I had a look around and could find some php config details, but didn't see anything that looked like what I think you might be asking about.

Agreed, any production web server would log every request.

I did check with the hosting company yesterday:

Am I right in thinking they are taken directly from the server logs, so they record every visit to the site?

Are there any circumstances under which a (human) visitor to the site wouldn't get recorded?

Indeed, the Webalizer engine uses the web server logs to generate its statistics. Each visit to the site is recorded by the web server in its logs, so there shouldn't be any visits missing from the statistics.

There are only two^* sane reasons I can think of:

1) Webaliser is failing to show you all the lines in the log

I've figured out how to access the raw server logs and can still only see my home IP and the ip of the webiste's servers. (I was only expecting to see one or two other visitors to access the site during the specific timeframe I'm interested in.)

2) This traffic is not actually hitting the webserver - for example they are getting the pages from a caching proxy

The website's only a few months old. When I installed wordpress I would have clicked the option to politely ask bots and spiders to go away. I've double-checked the Webalizer stats and the only other IP address is what will almost certainly turn out to be my Mum's (bots/spiders normally show up). A quick search in Google doesn't return any results either. I think I'm going to rule this option out.

* - another possibility is this is a load-balanced server and you're only seeing logs from one of the servers, but this is unlikely.

erk. No idea! I've not seen any mention of this in the help files I've been looking at recently. I've been using this hosting service for over 10 years and for the first few of those would have been looking at Webalizer and not noticing any strangeness. I think I'll file that under unlikely too.

[Kim]

If you're getting nothing in a known time period, then I think Occam's Razor has your answer.

An upstream caching proxy would still have to access the pages once.  You'd see that in the logs (perhaps you'd need to adjust the time period accordingly).  If for example you accessed the pages through the foobar.ac.uk caching proxy when you set it up, and then another person on a foobar.ac.uk machine accessed them later, it could get them from the cache. *BUT* any sanely configured cache will check to see if anything's changed on subsequent accesses after a while, and you'd see that request.

If it's being accessed over IPv6 your analysis tool might not understand v6 IP addresses, but it sounds like you've eyeballed the raw logs.

Load-balancer cockup... Possible, but unlikely, given this is your own hosting, and if there were load-balancing you'd be paying for it.

Malware retconning the logs to conceal its presence... Even more unlikely.

Struggling to think of other reasons... If the disk fairy ate the logfiles, you'd probably have noticed.  Could a backup have been restored over the top of them?

[Valiant]

If you're getting nothing in a known time period, then I think Occam's Razor has your answer.

An upstream caching proxy would still have to access the pages once.  You'd see that in the logs (perhaps you'd need to adjust the time period accordingly).  If for example you accessed the pages through the foobar.ac.uk caching proxy when you set it up, and then another person on a foobar.ac.uk machine accessed them later, it could get them from the cache. *BUT* any sanely configured cache will check to see if anything's changed on subsequent accesses after a while, and you'd see that request.

If it's being accessed over IPv6 your analysis tool might not understand v6 IP addresses, but it sounds like you've eyeballed the raw logs.

Load-balancer cockup... Possible, but unlikely, given this is your own hosting, and if there were load-balancing you'd be paying for it.

Malware retconning the logs to conceal its presence... Even more unlikely.

Struggling to think of other reasons... If the disk fairy ate the logfiles, you'd probably have noticed.  Could a backup have been restored over the top of them?

A lot of shared hosting comes with load balancing these days. But then you'd think caching services etc would all be setup properly.

[vorsprung]

Thanks all.

It's unlikely you'd be in a scenario where some and not all traffic is logged.

*nods* This is my understanding of the situation. Once I get to "highly unlikely" I can kick some ass.

what you haven't made clear is why it's a problem
it sounds like your website gets almost no traffic from what you are saying anyway

[Kim]

A lot of shared hosting comes with load balancing these days. But then you'd think caching services etc would all be setup properly.

Fair point.  I was thinking in terms of dedicated servers (virtual or otherwise).  And yes.

[Kim]

what you haven't made clear is why it's a problem
it sounds like your website gets almost no traffic from what you are saying anyway

I can imagine a scenario where you might put a document on the interweb, refer someone to that document, and then suspect that they haven't read it...

[Valiant]

Install Jetpack and see if their stats line up.

[Feanor]

Or, OTOH, you have put content online which you believe to be behind a login, and have reason to suspect that it's leaking somehow.

[vorsprung]

Or, OTOH, you have put content online which you believe to be behind a login, and have reason to suspect that it's leaking somehow.

If it's so top secret don't use wordpress

[vorsprung]

what you haven't made clear is why it's a problem
it sounds like your website gets almost no traffic from what you are saying anyway

I can imagine a scenario where you might put a document on the interweb, refer someone to that document, and then suspect that they haven't read it...

If it's important then email it to them.  Refer them to the website.  Print it out and post it to them.  If it isn't important leave it on a website

[nikki]

If you're getting nothing in a known time period, then I think Occam's Razor has your answer.
[...]
Could a backup have been restored over the top of them?

I'm feeling more comfortable calling it now I've accessed the raw log files.

They're read only and in a completely different place to the web back-ups. I trust the hosting company would be aware of it by now if one of their automated systems was over-writing stuff...

A lot of shared hosting comes with load balancing these days. But then you'd think caching services etc would all be setup properly.

okay, ta.

what you haven't made clear is why it's a problem
it sounds like your website gets almost no traffic from what you are saying anyway

Yes, sorry, deliberately so. Not a problem for me, but potentially a very serious one for someone else if the logs prove to tell a particular story.
The website was only ever intended to have 2 specific people (with the relevant log-in) read it.

Feanor and vorsprung: valid comments in the last couple of posts, but a different scenario here.

[Jaded]

what you haven't made clear is why it's a problem
it sounds like your website gets almost no traffic from what you are saying anyway

I can imagine a scenario where you might put a document on the interweb, refer someone to that document, and then suspect that they haven't read it...

If it's important then email it to them.  Refer them to the website.  Print it out and post it to them.  If it isn't important leave it on a website

You can lead a horse to water but...

Send them a link. It is entirely up to them if they read it.

[Pickled Onion]

Yes, sorry, deliberately so. Not a problem for me, but potentially a very serious one for someone else if the logs prove to tell a particular story.
The website was only ever intended to have 2 specific people (with the relevant log-in) read it.

If it's behind a log-in, proxy caching can be categorically ruled out. If they hit the log-in page it will be logged, if it's not in the raw logs you can be as good as certain they didn't. Given the seriousness of the allegation (if I'm guessing correctly) you can't be absolutely 100% sure there wasn't a gremlin that corrupted the log, but the chances of something like that affecting just that visit are very, very slim indeed.

[nikki]

If they hit the log-in page it will be logged, if it's not in the raw logs you can be as good as certain they didn't. Given the seriousness of the allegation (if I'm guessing correctly) you can't be absolutely 100% sure there wasn't a gremlin that corrupted the log, but the chances of something like that affecting just that visit are very, very slim indeed.

*thinks*
If they log in, then that information's got to be passed to the server, right? Basically a form submission and a corresponding redirection or something in response.

Re: guesswork. Yeah, there is probably enough in posts elsewhere to accurately guess what this is connected to...  *sigh*

[Pickled Onion]

Yes, exactly - if a log-in is required, only the server can authenticate and allow access, so the server will have logged it.

You have done a very thorough job at collecting the evidence, with statements from the ISP and everything else. Given the reputation for that profession to close ranks and the fact you'll only get one chance to present your case, I'd perhaps suggest you get someone to put together an "expert statement". Something laying out the facts as you have determined but signed as an independent authority.

[nikki]

Thanks Pickled Onion; that sounds like it would be a very prudent move.
I'll start asking around a bit for a suitable independent authority.