The Wednesday Update: Goodbye Gwiz

So welcome to another Wednesday update, stuff is actually starting to move along again!

Gwiz Update

So due to having too many projects on the go and lack of time, the Gwiz has been sold. I really enjoyed getting the car up and running but I couldn’t warrant the monthly cost (insurance wasn’t that cheap) and I can still do a self-driving project, just at a smaller scale!

Electronics Course

This has stalled a bit which is always bad when doing a course as you start forgetting what you have learned. However once I complete the main printing of the Inmoov robot, this will continue.

Inmoov Robot

So my main focus has been the Inmoov Robot and trying to fix the main lower structure that partially cracked. If this had gone (turns out it was far off) the robot would of fallen off the stand and smashed. That would not be a good day! I have also done some work on the arm which until the servo bicep holder broke, would of been complete

Turned out one of the main base plate was also damaged but this would be a complete rebuild, so I glued it.
This was printed in PLA which over time has been brittle
The orange bicep servo holder that snapped

Exercise

So I have started mountain biking quite a bit and have now got a nice (but old) full suspension bike. Today I also ran 3 miles on the treadmill and nearly died lol. I would like to get some fitness back as it helps me mentally with these sorts of projects.

The Wednesday Update: The Missing Edition

Hello and welcome to another Wednesday update, I know it’s been nearly a month since I did an update, and even that update was a bit thin with content.

The Kitchen aka the excuse

So nothing to do with robotics but the main reason I have done very little in the way of tech. My entire kitchen, dining room, and downstairs bathroom have been replaced including removing a wall which resulted in a lot of work, a lot of mess, and a lot of noise. This basically stopped me from doing anything interesting as after I would finish work on a Wednesday I was cleaning, moving stuff, or helping the builders who were friends of mine. They did an epic job however I think I will be cleaning for a very long time. At the same time, I also had my windows replaced.

Inmoov Robot

So a month on and it’s hard to remember what I was doing, I have made some progress with the Inmoov Robot which is good. I have nearly finished the second arm however one of the main bottom supports (which connects the robot to the chair pole) is now damaged which is not good! This will be quite a lot of work to sort out however if I don’t, the robot is likely to fall over and smash.

Gwiz

Nothing really has happened with this, I still want to see if I can make it drive itself.

Electronics course

I have made a bit of progress on this and passed the second test (barely due to the gap between lessons). I hope to complete this by next week

Hosting for Burf.co

So after some social media feedback to HostPapa, a new member of staff has been trying to help me lower the concurrency issues with my site (Entry Points). I still find this a terrible way to do business (unlimited everything as long as it comes in single-file) as that’s not the way the internet works. I did actually check CPU and Memory usage and it was barely anything!

The Wednesday Update: More Inmoov work

Welcome to another Wednesday update, as you can guess by the title, this afternoon was purely spent building the second arm/hand for the Inmoov robot. Many fingers broke in this process, they really are quite fragile.

I was making quite good progress up until the last finger broke and I ran out of fishing line which is used to control the fingers.

Once the arm is finished I hope to focus on the electronics and the programming side.

The Wednesday Update: The Wellness Edition

Welcome to another Wednesday update, due to Compsoft having a Wellness day (bloody epic if you ask me) I have had the whole day to do stuff…..

Gwiz update

So the new 100AH deep cycle Hankook DC31 batteries turned up and I would class it as a success, instead of being able to do about 2 miles on my second-hand batteries, I managed 10 miles without any issues or slowdowns. Everything seemed to work well, got up to 40mph (quite scary in a paper-thin car). I have also wired the car chargers into the original charging port of the car. Next really is to focus on making it drive itself.

Inmoov Robot update

So one of the main things I wanted to do today was fit the arms to my Inmoov humanoid. It still needs a lot of work, repairs, and some restoration however I would see this as a big step forward. One of the hardest bits to get right is setting up the biceps as they have a habit of going crazy and ripping the arm apart (I have 2 3D printers currently on printing replacement parts). I did manage to fit the left arm and part of the right arm however the right arm did break some parts. I hope to do a video soon.

Before I fired up the right arm
After I fired up the right arm

Misc

I am currently trying to work out how to control my electric wheelchair remotely, as mentioned this would be a great platform for an outdoors robot. I have also managed to fire up a 36v mobility scooter motor which could be quite good fun.

The Wednesday Update: The Sunny Edition

Welcome to another Wednesday update and boy has the weather been great the last 5 days. Compared to the usual rain we had for a good few weeks, the weather has been epic!

Gwiz update
Stuff has happened, the big thing being that I have now ordered brand new (yes actually new) batteries for it. I have gone with some 100AH deep cycle Hankook DC31. I will be running around 65 volts which does make the Gwiz shift a bit! They should arrive ready for the weekend ūüôā

Inmoov Robotic Head update

Well, this has started to look pretty cool now that I have tidied up the internals. I have mounted the amplifier, sound level PCB, and other things. I need to reprint the top part of the skull

I think I have got this to a great place. I do want to add a camera to it but to mount that onto the Pi will be a massive challenge. I also need to fix the Burf.co chat API as it’s dying badly. I would also like the Python script to autostart when the head is turned for all of the speech, TTS and API services.

Burf.co : Goodbye server!

So do not run random Linux scripts on a live server from the internet lol. I completely removed SUDO permissions from my box, 1.2 terabytes of data lost! It was a bad day, I was multitasking badly and paid the price. I managed to rebuild the server rather quickly (2 hours) however the main index data will take ages to get back. I like to see this huge mess up as an opportunity to do something different so watch this space!

Review

Let’s see how I am actually doing against the plan, I had a few lost sessions due to house improvements and covid, but how am I doing against the list

List from February

  • Math : Learn Algebra and kinematic (Little progress made)
  • Finish¬†Introducing Robotics¬† (Done)
  • Start and Finish¬†Robotic Vision: Principles of Vision¬†(Scrapped as very old)
  • Finish¬†Artificial Intelligence for Robotics¬†: This is the self driving course I really want to do but needs maths. (Not started yet)
  • Give a good shot to¬†Machine Learning¬†by Andrew NG : A very famous ML course (Not started yet)
  • Get Robotic head talking and listenering (Done, will go further)
  • Do something useful with Gwiz (Progressing)
  • Finish¬†Electronics course (Not started yet)
  • Start and finish¬†Self driving course using deep learning (Not started yet)
  • Build/use already built little robot to do maze solving work (Little progress made)
  • Finish Robotic Diploma (Missed off list but was important) (Done)
  • Finish Inmoov Robot (Missed off list but is very important) (Progressing)

I find it very odd how I missed the Robotics Diploma and finishing the Inmoov Robot (full size) off the main list as these are actually more important than most of the other things.

So 3 things done, 1 scrapped (it was mentioned in a previous post why), 2 processing well, and 5 academic courses to start. Seeing as it’s 3 months in, I think that is actually good progress.

The Wednesday Update: Bicep Edition!

So welcome to another Wednesday update, this one is rather special as I have actually managed to publish it on a Wednesday!

Robots Everywhere

I saw a video online of a Ukrainian kid who was learning robotics and had made a cool robot called Markobot. In the background of the video was an Inmoov robot, it wasn’t complete but it reminded me that I had one upstairs that I hadn’t finished yet and though I had been learning cool stuff and building epic things the last few months, nothing really compares to the Inmoov robot. For anyone interested in humanoid 3D printed robots, nothing comes near to the Inmmov Robot

Link to the article https://spectrum.ieee.org/video-friday-markobot

Inmoov Robot Progress Update

So I decided that the majority of today I should get back into fixing up this robot. One of the main bits that had not been assembled were the biceps, these are quite hard to configure and usually end up with broken parts. However, I had a rather successful day

I also had time to test out the hand

Wheelchair Fun

In other news, I managed to also get a broken wheelchair to kick back into life! I believe a great base for a mobile robot that needs to be used outside is an old wheelchair. I paid ¬£50 for this and it’s fantastic (I really enjoyed driving it around the garden). I plan to wire in a Raspberry Pi to allow me to remote control it! Then add some sensors, Lidar etc, and see if I can make it drive itself!

Misc

I am now on module 6 of my Robotics diploma! It’s getting pretty hard now (Inverse Kinematics was this module’s topic) but I hope to finish it!

The Wednesday Update: It’s alive!

Hello and welcome to another Wednesday update on a different day than Wednesday. This afternoon I tried to focus on 1 task, get my Inmoov robotic head talking. Short version, I DID IT!

As you can see there is plenty of room for improvement however it does work. I want to get the lipsync working a bit better (this is usually down to tweaking the volume output) and the voice is pretty terrible. I need to cure the speaker interference too.

However, there is something else going on here that’s pretty cool. It is wired up to the Burf.co Chat API which makes it a perfect testbed for the website. The API at the moment is just a standard implementation of an AIML library however this opens the doors to do some pretty cool stuff

Check out Wikipedia for more info on AIML

The code is written in Python and is fairly simple stuff, there are lots of examples to make your own Jarvis on the web. The pain was getting the Speech Recogniser working on the Pi. The LipSync purely works by the audio level coming from the headphone jack.

Misc

I also finished a course on Matlab to help me finish Introducing Robotics.

The Wednesday Update: The Speed Edition!

So another Wednesday update! This time, we have hit unprecedented speeds!

Gwiz Update: 65 volts of pure madness

Yup, I hit 34MPH in this bad boy uphill, I think any faster and I would need to change my pants. It does become apparent that maybe the brakes need upgrading! The next test is to take it out onto normal roads!

Inmoov head update

I have now started moving the electronics into the skull so that it is more self-contained. I personally think it’s looking pretty awesome. Once I have wired in the speakers I will be able to get it to talk and listen to commands.

Education

Still progressing with Introduction to Machine Learning however I wanted to focus on actually building stuff this week

Misc

I bought a cheap old robotic hover this week after the Rodney Brooks podcast that I recommended last week. I found it really interesting to see how it behaved and plotted its paths around my downstairs. It only has basic sensors (no Lidar) but it was really interesting. My son Max also found it rather interesting and spent 30 minutes playing with it.

Its been a while

So it’s been over 4 months since I last updated my blog, to me, it seems a lot longer, maybe that’s to do with the Lockdown? who knows?

So, what have I been up to? except drinking lots of Beer like most of the people under Lockdown are doing.  Well, I have been semi-productive, I would say pretty productive, however, compared to some of my friends (one who launched a radio station while on lockdown), I will retire to sem-productive.

Inmoov Robot

First 3 months of the year, the progress was fairly non-existent however due to the lockdown, I have come leaps and bounds.  I have rebuilt the head, neck, torso, fixed or improved tons of things, required most of it, and made a stand (which blow a hole in my ceiling and nearly killed me).  I am waiting for some potentiometer (Absolute pain to find the right ones) for the arms but I hope to have the biceps done soon.  I have made a start printing the legs for it via my new CopyMaster 400 3D printer which is pretty cool.

I have turned my conservatory into a robotics area especially for this robot which I have found has helped a lot.

Burf.co

So on the mission to learn C# and Azure, I have completely rewritten (all 20 lines of it) Burf.co Search Engine to be Azure Functions running in Docker Containers written in C#,¬† I actually really enjoyed doing this and again it’s come further along than the old Java/Kotlin one did in weeks versus months.¬† I have Azure functions that serve up the results to the website, parse and index websites, crawls sites and even a chatbot for fun ūüôā (SearchAI.uk)

I have even fired up the old HP DL580 server (currently keeping the house warm) to see if I can process data faster.

CommonCrawl WET File Processing

So I decided to try and write a script that would download the WET files from the CommonCrawl (56000 files, 8TB compressed).  These files contain 2.8 billion webpages or so and could be a really fun thing to process using ML etc.

Here is my V1.02 of this script, it’s hacky at best but its a start:

 

namespace SimonPlayGround
{
    class Program
    {
        private const string Path = "https://commoncrawl.s3.amazonaws.com/";
        private const string TargetFolder = @"z:\";
        public static int Jump = 2320;
        private const int Threads = 20;
        private const int Block = 10000;
        private const string UserId = "simon";

        public static async Task Main(string[] args)
        {
            var client = new WebClient();
            var paths = new List<string>();

            var mongo = new MongoClient(new MongoClientSettings()
            {
                Server = new MongoServerAddress("192.168.0.150"),
                MaxConnectionPoolSize = 500
            });

            var db = mongo.GetDatabase("WEB");
            var collection = db.GetCollection<Page>("wet");

            DownloadPaths(client);

            foreach (var line in File.ReadLines(TargetFolder + "wet"))
            {
                paths.Add(line);
            }

            // hack to remove done ones
            for (int i = 0; i < Jump; i++)
            {
                paths.RemoveAt(paths.Count - 1);
            }

            var tasks = new List<Task>();
            tasks.AsParallel();

            for (var i = 0; i < Threads; i++)
            {
                var filename = paths.Last();
                tasks.Add(Task.Run(() => Process(filename, collection)));
                paths.RemoveAt(paths.Count - 1);
            }

            while (tasks.Any())
            {
                await Task.WhenAny(tasks);
                var finishedTasks = tasks.Where(t => t.IsCompleted).ToList();
                foreach (var finishedTask in finishedTasks)
                {
                    tasks.Remove(finishedTask);
                    if (paths.Count > 0)
                    {
                        var filename = paths.Last();
                        tasks.Add(Task.Run(() => Process(filename, collection)));
                        paths.RemoveAt(paths.Count - 1);
                        Console.WriteLine($"Left {paths.Count} {tasks.Count} {Jump}");
                        // todo write here the number of files done
                    }
                }
            }
        }

        public static async Task Process(string filename, IMongoCollection<Page> collection)
        {
            var file = await DownloadWetAsync(filename);

            await ParseWet(file, collection);
            Console.WriteLine($"FILE PROCESSED");
            File.Delete(file);
            Jump += 1;
            // todo write here that file was completed
        }

        public static async Task ParseWet(string filename, IMongoCollection<Page> collection)
        {
            using StreamReader sr = File.OpenText(filename);
            string s;
            StringBuilder sb = new StringBuilder();
            var foundDoc = false;
            var foundURL = false;
            var url = string.Empty;
            var count = 0;
            var pages = new List<Page>();

            Console.WriteLine($"Processing {filename}");

            while ((s = sr.ReadLine()) != null)
            {
                if (foundDoc == false && s.Equals("WARC-Type: conversion"))
                {
                    sb.Append(s + Environment.NewLine);
                    foundDoc = true;
                }
                else if (foundDoc == true && s.Equals("WARC/1.0"))
                {
                    var from = sb.ToString().IndexOf("Content-Length: ", StringComparison.Ordinal) + "Content-Length: ".Length;
                    var text = sb.ToString()[@from..sb.Length];
                    var body = text.Substring(text.IndexOf(Environment.NewLine, StringComparison.Ordinal) + 1);
                    foundDoc = false;
                    foundURL = false;
                    sb.Clear();

                    try
                    {
                        count += 1;
                        pages.Add(new Page()
                        {
                            Url = url,
                            Body = body
                        });

                        if (count % 1000 == 0)
                        {
                            Console.WriteLine($"Procsessed {count} {DateTime.Now}");
                        }

                        if (count == Block)
                        {
                            count = 0;
                            await BulkSave(pages, collection);
                            Console.WriteLine($"{Block} done {DateTime.Now}");
                            pages.Clear();
                        }
                    }
                    catch
                    {

                    }
                }
                else if (foundDoc == true)
                {
                    sb.Append(s + Environment.NewLine);

                    if (foundURL == false && s.StartsWith("WARC-Target-URI: "))
                    {
                        var from = s.IndexOf("WARC-Target-URI: ", StringComparison.Ordinal) + "WARC-Target-URI: ".Length;
                        url = s[@from..s.Length];
                        foundURL = true;
                    }
                }
            }

            // save any left over
            if (pages.Count > 0)
            {
                await BulkSave(pages, collection);
            }
        }

        public static async Task BulkSave(List<Page> pages, IMongoCollection<Page> collection)
        {
            try
            {
                var updateOneModels = pages.Select(x =>
                {
                    var filterDefinition = Builders<Page>.Filter.Eq(p => p.Url, x.Url);
                    var updateDefinition = Builders<Page>.Update.SetOnInsert(p => p.Body, x.Body);

                    return new UpdateOneModel<Page>(filterDefinition, updateDefinition) { IsUpsert = true };
                }).ToList();

                var resultWrites = await collection.BulkWriteAsync(updateOneModels);
                Console.WriteLine($"OK?: {resultWrites.IsAcknowledged} - Inserted Count: {resultWrites.InsertedCount} {resultWrites.ModifiedCount}");

                updateOneModels.Clear();

            }
            catch
            {

            }
        }
        public class HttpRetryMessageHandler : DelegatingHandler
        {
            public HttpRetryMessageHandler(HttpClientHandler handler) : base(handler) { }

            protected override Task<HttpResponseMessage> SendAsync(
                HttpRequestMessage request,
                CancellationToken cancellationToken) =>
                Policy
                    .Handle<HttpRequestException>()
                    .Or<TaskCanceledException>()
                    .OrResult<HttpResponseMessage>(x => !x.IsSuccessStatusCode)
                    .WaitAndRetryAsync(10, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)))
                    .ExecuteAsync(() => base.SendAsync(request, cancellationToken));
        }

        public static async Task<string> DownloadWetAsync(string line)
        {
            var filename = line.Split('/').Last();

            if (!File.Exists(TargetFolder + filename))
            {
                Console.WriteLine($"downloading {filename}");

                using (HttpClient client = new HttpClient(new HttpRetryMessageHandler(new HttpClientHandler())))
                {
                    using (HttpResponseMessage response = await client.GetAsync(Path + line, HttpCompletionOption.ResponseHeadersRead))

                    using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
                    {
                        using (Stream streamToWriteTo = File.Open(TargetFolder + filename, FileMode.Create))
                        {
                            await streamToReadFrom.CopyToAsync(streamToWriteTo);
                        }
                    }
                }
            }
            else
            {
                Console.WriteLine($"GZ exist {filename}");
            }

            var wetFile = TargetFolder + filename.Substring(0, filename.Length - 3);

            if (!File.Exists(wetFile))
            {
                Console.WriteLine($"Decompressing {filename}");
                DecompressGZip(TargetFolder + filename, wetFile);
            }
            else
            {
                Console.WriteLine($"WET exist {wetFile}");
            }

            return wetFile;
        }

        public static void DownloadPaths(WebClient client)
        {
            client.DownloadFile("https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2020-16/wet.paths.gz", TargetFolder + "wet.gz");
            DecompressGZip(TargetFolder + "wet.gz", TargetFolder + "wet");
        }


        public static void DecompressGZip(String fileRoot, String destRoot)
        {
            using FileStream fileStram = new FileStream(fileRoot, FileMode.Open, FileAccess.Read);
            using GZipInputStream zipStream = new GZipInputStream(fileStram);
            using StreamReader sr = new StreamReader(zipStream);
            var data = sr.ReadToEnd();
            File.WriteAllText(destRoot, data);
        }
    }

    public class Page
    {
        [BsonId] public ObjectId Id { get; set; }
        [BsonElement("url")] public string Url { get; set; }
        [BsonElement("body")] public string Body { get; set; }

    }
}