November 2, 2018

Windows 10 vs. Helvetica

More than three years after general availability of Windows 10 we finally rolled it out to the users. Well, one reason we didn't roll it out faster was because the big bosses wouldn't buy new computers, so some users got stuck with old computers running Windows XP for ages.

Immediately after setting up the brand spanking new desktops, users came crying that their ERP reports look different than before. I started explaining to them that it's normal since every version of Windows had different fonts and drivers, so even if the same fonts were used it could look slightly different, especially if it was a big jump from XP to 10.

But this time the users pointed out to me that the fonts looked completely different. I looked again and sure enough, printouts from Windows XP had serif fonts, while printouts from Windows 10 were sans-serif. Hmm, this suggested that it could be a font substitution problem. I examined the fonts closer and realized the serif fonts from XP were Tahoma, while Win 10 used Arial.

Now the problem got interesting. There shouldn't be any reason for a type change from Tahoma to Arial, especially if it was a font substitution issue. Upon further inspection of the reports, I realized the ERP reports were all created with the default font Helvetica. I looked at Windows' font substitution setting, and sure enough, Arial is the standard substitution font for Helvetica.

So there lies the big mystery, this suggests that for the past 10 years, all our Windows XP and Windows 7 computers have been substituting Tahoma for Helvetica. While Windows 10 actually fixes the issue and correctly uses Arial? And now I have to put in the wrong value so users get to use the wrong font?

October 31, 2018

Milestone Pod

I'm not a very fast runner so all the new-fangled running dynamics are wasted on me. However, I like having foot pods on my shoes, especially now that I have a new watch that supports linking to multiple sensors of the same type. Meaning I can have one foot pod on each pair of shoe I have. The reason I like foot pods is to get real time pace. As I just said I'm not a fast runner, so getting pace from my watch instead of running "by feel" is much more useful for me.

However, I was no longer able to get cheap foot pods locally, and as the running fad boomed, running accessories got even more expensive than before. But while out looking at running shoes one day, I came across the Milestone Pod. I was a little surprised since I had never heard of the Milestone Pod before seeing it in that store. The box seemed to suggest that it needs a dedicated app, but a quick online search revealed that it can be used as a realtime foot pod using Blueooth. For reference the Milestone Pod is only 1/3 the price of the Garmin SDM4 foot pod locally.

I decided to let my SO have a go at the Milestone Pods since the app is so pretty and has automatic shoe mileage tracking. Unfortunately, in real life the Milestone Pod doesn't work so well with her Fenix 5S. I know the 5S has trouble with sensors, but we never had issues with the older ANT+ foot pods. And some of the trouble I see online seem to suggest that it's due to distance. She's not that tall so I didn't think it would be a problem. Actually, at a indoor track where we often run, the pod wouldn't connect at all. It's probably due to interference from the Bluetooth lap counting system they use, but it's just completely disappointing and I wish I had bought ANT+ foot pods instead.

After a lot of fiddling I decided to swap my ANT+ foot pods with her and I took both of the Milestone Pods for myself. I have the Forerunner 935 which works somewhat better, but I notice even with the 935, a lot of times I look at the pace while running the field is showing blank, and a lot of times after the workout I find the cadence field to be erratic.

Milestone Pod's support people suggest the reason I was having issues was because I used both the Milestone Pod app and my Garmin watch to calibrate the foot pods. They suggest that I should disable the watch's calibration and set the calibration factor to 100.0, and only use the Milestone Pod app to calibrate the pods. Unfortunately, I find the MP app to be unrealiable and it would keep going back and forth between calibration values. On one run it would be way too fast, but after calibration the next run it would be way too slow, rinse and repeat. Finally I decided to delete the app entirely and only rely on the my watch's auto-calibration, which works much, much better for me.

Update: I kept having more and more issues with cadence on the Milestone Pod, until finally the cadence would completely disappear when I pause. However, the Milestone app still records the cadence perfectly. This time the MP support people suggest that the battery may be too low for broadcasting BLE, even though it's still reading medium-high in the app. I decided to change the battery and sure enough, that fixed the cadence problem. I remembered reading somewhere that the MP can eat through batteries, so I checked the official specs and it indeed says four to six months. Also places like Zwift support says battery on the MP may be enough for other devices, but may be too low for Zwift.

So yes, I'm tentatively in love with the Milestone Pod again. But let me go out for a few more runs and see if the new battery works out.

September 25, 2018

Join Windows 10 to Windows 2000 domain

We bought new computers that only support Windows 10. I tried to install Windows 7 on them but it didn't work very well, besides, we could no longer get Windows 7 licenses, so we finally started rolling out Windows 10 to the users. I immediately ran into issues joining them to the old Windows 2000 domain. Google search returns Windows 10 no longer supports Windows 2000 domains and old servers need to be upgraded. Hmm, I distinctly remember joining Windows 10 computers to the domain since some managers have had new computers since last year, it's only the users are getting Windows 10 right now.

After more searching I found sites saying that security policy needs to be modified. I also don't remember doing anything like that previously. I checked my notes and I had nothing on issues with Windows 10. After some brain wrangling I figured that it must be something about SMB. It turns out that SMBv1 is not installed by default only on newer versions of Windows 10 as described in this article. We did new installs using 1803 and SMBv1 is no longer included which was why it worked last year and no longer works now.

Adding SMBv1 back is just a matter of turning on a Windows feature, and after that Windows 10 can be joined to the Windows 2000 domain without issues.

End note: yes yes, I know that old servers should be upgraded, we have some new servers but they've yet to be put into production mostly because old programs need a lot of time to be ported. I hate it when people tell me old hardware or software are no longer supported, just upgrade.

September 21, 2018

Siemens PG 720 PII

The Siemens PG series was specialized notebook with a special port for connecting to their PLC devices. Many years ago I asked Siemens whether we could use a regular notebook and buy just the special PLC adapter, they said it was possible and is actually recommended. However, the the purchasing people said we're rich, and Siemens notebooks are very rugged and should last much longer than regular notebooks, so we ended up buying a PG 720 PII for a lot of money.

And rugged they are! I got called in recently to support the PLC, and was surprised to find that the PG still worked.

This is what a new PG 720 P looks like, picture found on the net. The PII is identical except for better hardware specs. Oh, one special thing about the PG 720 PII we have (not sure about the P) is that it has a 2.88 MB floppy disk drive with laser tracked heads so it was able to read any floppy disk we throw at it. Back when we used a lot of floppy disks I used to borrow the PG just to read the bad disks.

It has a keyboard that also works as the screen cover, ours was long broken so it was just unplugged and replaced with regular mouse and keyboard. The LCD screen still works but is cracked and faded, so we use an external monitor. The plastic around the screen was all brittle. I wanted to move the PG a bit for a better angle to photograph and another huge chunk of plastic broke off. The engineers yelled at me to get my hands off his precious. Good times.

September 13, 2018

ARP cache poisoning attack

After years of putting in purchase request after purchase request for an antivirus program. It took a ransomware attack for my bosses to see the light, and we finally bought ESET Endpoint Security. ESET (NOD32) has a pretty bad reputation locally for being the antivirus product that you install if you want to be infected with a virus. I'm not sure for the reason why it was so badly rated, but perhaps because everyone was running pirated versions, and maybe the pirated copies don't actually work, but that's another story for another time.

Of course, we looked at several competing products, and I almost decided on Bitdefender, but during the 30-day trial period I discovered that Bitdefender was really slow if I was scrolling through my files in Windows Explorer with my cursor keys. I asked the support people and their clever answer was: don't move around with the cursor keys, just use the mouse and click on the file you want to use. So I put in a request and bought ESET Endpoint Security.

The morning after I received the licenses in email, boss called and asked if I've finished installing the antivirus. (Oh, there's another story involving the email, which is also another story for another time.)

"Finished? I've just started downloading the setup files, and I have to setup the management server."

"Good, so after that every computer will have the new antivirus?"

"Uh... not exactly, we have to set up the management server, create policies and exceptions, then deploy the client software to the domain, shouldn't take more than a week or two. But we have some really old computers that are being replaced next month, so we'll hold off installation for those after they're replaced."

"Good, finish installing everything by noon and sign off the project."

"What? It's not a home product that I just click on setup and use the defaults, we have hundreds of computers in different configurations and it will take at least a few days to get all the policies correctly, and we must test the policies then do a phased rollout so we don't run into too many problems all at once. Plus we have to remove the old antivirus which have to be done manually since you won't buy a commercial product and all we used were free home versions."

"That's why you have your staff to help you. I'll give you some extra time, just finish everything by today."

So I booted up the management server, created a default policy, let my guys go around the office uninstalling the old antivirus (if any) and rolled everything out all at once. Less than a minute later my phones started ringing off the hook and the management server started reporting ARP cache poisoning attack. Users were reporting pop-up menu complaining ARP something something, they can't access the ERP, and they couldn't print to an old cheap printer shared from one of the computers.

Printing to the shared printer was easy, just a matter of adding an exception to the firewall, but it took longer than it should, since I was unfamiliar with ESET, and it took me more a few minutes to find the option and deploy the updated policy to the clients.

ARP cache poisoning attack was less obvious. I looked and it was coming from the SQL database server (which also explains why access to the ERP was blocked). Hmm, strange, then I realized the database server has a NIC team. Aha, some doc reading led me to adding an entry to IDS exception setting. (The ESET support pages said to add entries to Trusted zone and Addresses to be excluded from IDS didn't work at all. The entry had to be added to the Network attack protection settings page.) After that, no more ARP cache poisoning attack messages, and then I started getting warnings of duplicate IP address on the network.

Okay, add another entry to the IDS exception. Yay! All done, except the shared printer still didn't work. It's probably another non-obvious setting which I'll figure out another time.

May 4, 2018

I'm a sysadmin, not a miracle worker

Quite often, I get calls from friend's friend's friend's friend asking for computer help. Not long ago, someone called me up to ask for help in setting up osCommerce. Like I mentioned previously, I'm not a web developer and was never interested in web e-commerce. In fact, at the time I had never even heard of osCommerce. But the guy said, "I heard you're good in computers! I'm sure you can fix the problem!"

Uh... so I decided to take a look at his site. Fortunately enough, the problem turned out to be a permission setting, and was actually mentioned in the installation FAQ on the osCommerce site, and I solved his problem in about five minutes using the permission tool on his web host.

A few weeks later, he called me up again and asked for help in moving a website from one web hosting service to another. How hard could that be? I thought it would only involve copying some files, and maybe updating the DNS. But noooo, the new web hosting service had already done all that for him. The problem turned out to be a hard coded URL reference in the MySQL database. Of course, up to that point, I had never used phpMyAdmin or touched a MySQL database, but I still managed to fix it for him by changing the value in the database.

Next. By this time, I had already become friends with this guy, and when I showed him my Google Apps site, he wanted to do the same for his domain. So I helped him set things up, but a few days later, he called and said that the contact form on his website couldn't send messages to Google. I had to modify the PHP code in the site, without knowing one bit of PHP.

(By the way, since becoming friends with him, I realized he is a web developer and a consultant.)

Not long after that, he called me up again asking about flashing firmware for cellular phones. Turned out he was importing those "shanzai" imitation phones from China, and needed to re-flash the firmware for local use. Flashing firmware was something I do know how to do, so he sent me a phone, a flashing cable, and a single .bin firmware file. With no instructions and no programs. I had to figure things out by first doing an image search for that particular phone, then browsing all the Chinese websites about hacking phones for some clue.

I ended up trying so incredibly many things in order to get it to work, until the battery ran out, and I realized he didn't even bother to give me a battery charger! When I finally figured out how to flash the phone correctly, the screen didn't work after the phone rebooted. It turned out that I had to choose the type of LCD screen the phone has. Since there were no docs whatsoever, and the screen was already not working from the bad flash, I ended up trying every possible combination in the flash program, until it worked. Pure luck I didn't turn the phone into a brick!

I did all these thanks to Google, but he thinks I'm GOD.

April 16, 2018

Ultimate Ears Wonderboom

I've wanted to buy new Bluetooth speakers for ages. I rarely have time to sit down to listen to music, but I want something to listen to while working out. Ultimate Ears lowered the prices of speakers recently, so I went online to check them out. The Wonderboom Patches (Freestyle Collection) one caught my eye with the bright colors, and after checking out online reviews I ordered one immediately.

Unfortunately, what the reviews didn't talk about was that colors of the actual Wonderboom is nothing like the colors in the pictures. The colors of the actual product are faded and to me it looks like dirty jeans with printed pictures. I took a picture of the actual speaker and... it turns out that the colors look much nicer in the photograph for some reason. Normally photos will look faded compared to real life, but in this case, the colors stood out and it looks wonderful compared to the real thing.

Yeah, the colors are disappointing but luckily the speaker sounds awesome for the small size and it's helping greatly with my workouts.

March 15, 2018

Arduino RFID (Part 5 The Final Chapter)

(... continued from part 4)

As implemented, my Arduino RFID boxes are in the style of separate In and Out walkways as seen in most typical MRT systems where the incoming and outgoing people use completely different walk ways and separate scanners. Unfortunately we don't have electric gates, so sometimes employees get confused and scan their cards multiple times at the wrong scanners. My Arduino code prevents multiple scans that immediately follow one another, and our Windows side software will take care of other duplicate scans.

The guards have a special RFID card that's used to "swap sides". Our In and Out walkways are normally fixed, but in case a box goes out of order, we can immediately change the walk ways and redirect the employees to scan correctly. So far we've only had to do this a couple of times, when the POE Ethernet Shield blew up, and when the electricity went out for so long that the UPS that feeds the POE switch ran out of power, and we had to rig up a battery pack for a single RFID box to let the employees get off work.

The point of this is that the RFID card itself doesn't know if it's "inside" or "outside" of the premises. The boxes only do the scanning and logging, and it's up to the Windows software to figure out the working hours and overtime hours. The Windows software can also check the current time and check the logs to see if the employee is "inside" or "outside". I mentioned in the previous article (also via source code), the employee's name and code are written onto the card. They're displayed when the card is scanned for the user to read, but only the RFID card's UUID and the employee code are logged. There's no need to log the name.

However, since the MiFARE Classic cards are actually capable of being written to (like cash cards). An alternate design is the bus system where a single scanner can be used for both incoming and outgoing, and it's up to the card itself to remember its state. In this design, if the employee forgets to scan the card, the in/out status would be reversed, and they would need to reverse the status by scanning the card again, or letting HR reverse the status. This was to be my initial design, but I felt the employees would be too confused by this implementation.

Oh yeah, while working on the RFID boxes, one issue that came up was the accuracy of the time clocks. I tried to explain to the HR people what NTP is, but couldn't get through their skulls. Eventually I ended up doing a completely different project to handle the clocks.

All good things must come to an end, it's been a while since the last article, and over the years my Arduino RFID boxes have performed admirably, except for every month or two (roughly once every 10,000 scans) HR would complain that an employee missed a scan which I could never discover the problem. So late last year (2017) I finally gave up and the company purchased new scanners directly from vendors with warranty and we switched to the new system on the first day of 2018. And two and a half months into this year we've already missed more than 20 scans. Garbage I say.

However, since Raspberry Pi 3 B+ is released today, I might pick up this project again just for fun. The Raspberry Pi is easier to manager than the Arduino because it runs a full OS and even though I've never written about it here, I've played with them quite a bit since the very first release and I've done a really simple project that I'm actually proud of. Will write more later.

March 14, 2018

RIP Hawking

"The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge." 

I don't think it's an exaggeration to say that Stephen Hawking was every geek's childhood hero. Rest in peace, and thank you for being an inspiration for a generation of scientists and engineers.

February 23, 2018

Fake Logitech Wireless Presenter R400

I've been looking to buy a Logitech Wireless Presenter R400 since my ex-boss stole mine. The R400 is like the pinnacle of the PowerPoint remote since it does everything so well without any fuss, and it's backed by a well-known brand. I briefly considered again getting the R800 with green laser and vibrating alarm, but the few times I used green laser pointers in my presentations, the overly bright green laser only served to annoy my audience.

Anyway, if you've looked at online shops selling the R400, you might've noticed there are tons and tons of cheap "genuine" R400's. I found a shop selling one slightly cheaper than retail, and it showed the packaging with the Logitech logo, and it has hundreds of feedbacks claiming the shop is really good and the product is genuine. All the signs point to it being fake, so I bought one immediately.

Well, I couldn't lose anything. If it's fake I'll just return it, if it's genuine I've got myself a good deal, so here it is. Oops, I forgot to take a picture without the outer wrapper, but it says Logitech R400 on the box, and it has the old Logitech logo. Hundreds of online shops show this genuine blister packaging. Inside the plastic shell it still looks fine, but once unwrapped...

Comparing to a genuine R400. All images has the genuine on the left and fake on the right. I lost my original R400 when my ex-boss stole it. The one pictured in this post is a new one I just bought from an authorized dealer at list price after the fake one turned out to be, well, fake. Click the images to see close-ups.

The genuine one has better plastic and all the silk screening on the buttons as well as the buttons themselves are well-aligned. The genuine one has somewhat shinier plastic on the buttons, but the rest of the body feel similar.

As said above, the bodies feel quite similar, so that's a good thing for the fake one if you just want a cheap PowerPoint remote and don't care much for quality. There are many really cheap "R400-style" presenters being sold without the Logitech logo, but this is being sold as a Logitech R400, so I'm calling it a counterfeit. Notice my genuine one has the new Logitech logo that they started using around mid-2015.

The laser covering of the genuine one is curved and shaded. The fake one is flat and clearer so you can see the hole behind the covering.

The power slider of the fake one is really difficult to slide, and the shape of the switch doesn't quite fit inside the cutout so you can see a bit of green there. On the fake R400, the red sticker that's supposed to be shown when the remote is off is missing.

The battery compartment look similar except for the text, but the springs are lower quality on the fake one, and the batteries don't sit so well. Oh, and the fake one doesn't come with batteries. The genuine one has the newer R-R0008 part number while the fake one has the old R-R0004 part number. The fake one is missing a serial number.

The battery covers look similar, but the genuine one is shiny inside while the fake is matte. The battery cover of the fake one doesn't fit quite well and will fall inside if pushed too hard. Funnily enough the two battery covers are incompatible with each other.

The USB receivers look very similar, but the genuine one is shinier. The fake one appears to have markings that look used, but it came that way.

They come with different storage bags. This is just a difference with the year of production. Older R400's also came with the pouch on the right side, while newer ones have the low cost cloth pouch. The fake R400 comes with the old style pouch, but it feels rough compared to the genuine one that my ex-boss stole.

And finally, this is what a current genuine Logitech R400 packaging looks like. Even though the Logitech website still shows the old logo, but they're being sold with the new logo and new style packaging. And instead of being called wireless presenters, they're now called laser presentation remotes.

February 14, 2018

Can't boot into Windows after BIOS update

I think the scariest thing for a sysadmin is Windows failing to boot, especially after an unnecessary (?) BIOS update. Recently we got two Lenovo ThinkServer TS150's. They're cheap and work really well, and we quickly put one into production since the old server just suddenly up an died. So I was staring at the other server sitting there doing nothing except having a brand new install of Windows Server 2016, and seeing that the BIOS is dated 2016 and the latest is 2018, so I decided to upgrade the BIOS.

After the flashing was completed and the system rebooted, I got a black screen and this text: Error code 1962 - No operating system found. Well, the BIOS must've been reset to defaults. Go into BIOS, nope, everything is same as before. Maybe there are new BIOS options... tried some different configurations... didn't work... tried every possible configuration... didn't work... tried resetting to defaults... tried optimized setting... tried swearing... nothing worked. Luckily I have the other TS150, but since it's in production I had to wait until midnight to take it offline to look at the BIOS. Compared all settings and they're all identical except for the BIOS dates. Tried all possible configurations again just in case I missed something.

Next, I tried using Linux Live CD's, Windows 10 Live DVD's, and also Window' own rescue mode to flash different versions of the BIOS. Nothing worked. Tried going back to the original BIOS dated from 2016 but it wouldn't let me. Apparently there was a security update in 2017 and they disabled going back to older versions.

Next, I used rescue mode and the bootrec command to tried to fix the boot sectors. No go.

Gave up. Tried installing Windows from scratch. Nope, Windows complains that it can't be installed to this disk because the hardware may not support booting to this disk. Nooooooooooooooo.

Punch reset button in frustration. So while I was tearing my hair out again and pondering what to do next, I suddenly saw the familiar screen.

And next thing I know, Windows Server 2016 was booted up like nothing has happened.

After much head scratching, I discovered the reason it booted was because it was booting from the Windows install DVD, and because I was tearing my hair out and ignoring the server, the "Press any key to boot from CD or DVD..." prompt timed out, and it automatically booted into Windows. Nothing was changed in the BIOS, the boot sequence was correct, and after testing I confirmed that it will only boot Windows if it was booting from the DVD initially then let the prompt time out.

So... apparently after any BIOS update, something somewhere got modified in the boot sector and it would no longer boot correctly. But booting from the DVD then letting it time out seemed like a really strange thing, since this suggested the hard drive's boot sector was still functioning properly, it just wouldn't boot as the first boot device. Tried searching Google for this problem and found thousands and thousands of people with similar problems and no real fix except things I've already tried. Most ended up reinstalling Windows, which didn't work for me.

Well, after a week of even more head scratching, I finally came up with a working and reproducible solution (workaround). The ThinkServer came pre-configured with two hard drives which are configured as RAID1 array using the onboard Intel RSTe. I'm guessing the problem could be related to the Intel chipset and the RAID array configuration, but a BIOS update should not mess it up so much. Anyway, the fix was to remove one of the hard drives from the RAID array by booting into the Intel RSTe configuration screen and selecting the option "Reset Disks to Non-RAID". Remove one of the drives then add it back immediately. After that reboot into Windows using the DVD workaround method above. After the array was re-built, Windows could boot normally again.

Now let me go test this on my production server.