Mysql – Missing DB entries

MySQLPHP

I have run into a problematic situation.

I oversee the administration of 5 php web pages on one server, each having their own database. All pages work like interactive forms drawing and storing data into the DB. They all have the same structure but with different variables and screen outputs.

A few days ago between 22:00 and 06:00 (exactly the night when the leap second was applied), one of the web pages did not store any entries into its DB. All the other worked fine. From the error and access logs in apache it looks like that during that night, there was no traffic from the webpage at all, so it looks like a connection problem. The users report that everything was working fine which we later confirmed with camera footage and also several temp files on their computer. These are rests of generated data that was sent into their printer.

I tried to check the cable and it is ok. Page also informs the user if there is a problem with the DB. And I do not think that they could work from cached site data for all night long since there would be problems with the DB connection. The data was not deleted because the auto-increment ID's are ok. I also thought of the leap second but all pages use the same date time function and there were no other problems reported after it was applied. Also data was missing even before the leap second was applied. After 06:00 the storage of new data functioned as normal while there was no one that did anything either with the site or the database.

Me and my boss have run out of ideas what could be wrong. Anybody experienced something similar?
And some server info:

Debian 6.0.6
Apache/2.2.16 (Debian)
MySQL client version: 5.1.66

If you need any more information feel free to ask.

Thank you

Best Answer

Weird behaviour.

sql

Just because the auto increments are "ok" it does not mean the data has not been deleted. According to https://stackoverflow.com/questions/970597/change-auto-increment-starting-number it can be changed via alter table.

traffic

That there were no traffic by the webpage is weird as users worked with the website. Is it possible that the server could not be accessed and another server were uses during the time?

got hacked?

I would check wether someone connected via ssh or so and did some bad stuff via shell. Also check your system for malware.

other local scripts/programmes

I don't think this the reason but for completeness I added that.

Do you have any script or programme running that does sth. with sql and could have caused this (e.g. bug)?