I've built a file copying routine into a common library for a variety of different (WinForms) applications I'm currently working on. What I've built implements the commonly-used
CopyFileEx method to actually perform the file copy while displaying the progress, which seems to be working great.
The only real issue I'm encountering is that, because most of the file copying I'm doing is for archival purposes, once the file is copied, I would like to "verify" the new copy of the file. I have the following methods in place to do the comparison/verification. I'm sure many of you will quickly see where the "problem" is:
Public Shared Function CompareFiles(ByVal File1 As IO.FileInfo, ByVal File2 As IO.FileInfo) As Boolean Dim Match As Boolean = False If File1.FullName = File2.FullName Then Match = True Else If File.Exists(File1.FullName) AndAlso File.Exists(File2.FullName) Then If File1.Length = File2.Length Then If File1.LastWriteTime = File2.LastWriteTime Then Try Dim File1Hash As String = HashFileForComparison(File1) Dim File2Hash As String = HashFileForComparison(File2) If File1Hash = File2Hash Then Match = True End If Catch ex As Exception Dim CompareError As New ErrorHandler(ex) CompareError.LogException() End Try End If End If End If End If Return MatchEnd FunctionPrivate Shared Function HashFileForComparison(ByVal OriginalFile As IO.FileInfo) As String Using BufferedFileReader As New IO.BufferedStream(File.OpenRead(OriginalFile.FullName), 1200000) Using MD5 As New System.Security.Cryptography.MD5CryptoServiceProvider Dim FileHash As Byte() = MD5.ComputeHash(BufferedFileReader) Return System.Text.Encoding.Unicode.GetString(FileHash) End Using End UsingEnd Function
CompareFiles() method checks a few of the "simple" elements first:
But, you guessed it, here's where the performance takes the hit. Especially for large files, the
MD5.ComputeHash method of the
HashFileForComparison() method can take a while - about 1.25 minutes for a 500MB file for a total of about 2.5 minutes to compute both hashes for the comparison. Does anyone have a better suggestion for how to more efficiently verify the new copy of the file?
Do we need to encrypt and sign the payload to prevent tampering and ensure integrity ?
Assuming we have JWT bearer in the header and the API is secured with HTTPS.
Given an example of a payload that will be sent to the API that changes data of a user profile.
Do we need to do a md5 ( payload + private key ) for a signed payload ? Example:
md5(namebobage10genderm_private_key)md5 code = fdd5a4a41fc0ab84d4792fa8b08d8e17
The new payload would be
When the server receives the API call , it will also do the md5 encryption of the payload and compares the signed value in order to ensure integrity.
Please let me know your thoughts regarding this and do we really need this ? As we already have HTTPS for the API and JWT to authenticate the user calling the api.
This is quite subjective , the api call is still vulnerable to man in the middle attack , where the api call can be intercepted via proxy before it hits HTTPS . As long as the attacker does not know the algorithm, whatever data being modified during the intercept will be rejected at the server as a signed payload was sent alongside with the payload to verify at the server end.
Please enlighten me . Thank you
Is there a way to disable checksum with active storage? I am migrating from paperclip to active storage, using amazon s3 storage. There is no way to get the get the md5 checksum encryption code without communicating with the bucket, which is too time consuming to migrate millions of attachments in our production database. The checksum column has a null: false constraint, and removing this raises errors in the migration and in uploading the file that seem to be unavoidable. I feel I must be missing something here, is there any way to avoid using the checksum column?
As I'm writing code to install on a target machine, I was wondering about the dependencies and noticed that there were no openssl library needed. I wondered because I know I am using OpenSSL:
#include <openssl/md5.h>...MD5(a, b, c);...
To my surprise, it seems that we only get linked against libc. Is MD5 really implemented in libc and not in some libssl library?
objdump gives me the info about the linked library:
Dynamic Section: NEEDED libQtCore.so.4 NEEDED libstdc++.so.6 NEEDED libgcc_s.so.1 NEEDED libc.so.6 SONAME libcontent.so
As suggested by noloader I tried with ldd and still fail to see a library that would make sense for MD5. libcontent.so is directly using MD5()...
ldd ../BUILD/snapwebsites/plugins/content/libcontent.so linux-vdso.so.1 => (0x00007fff4f3ff000) libQtCore.so.4 => /usr/lib/x86_64-linux-gnu/libQtCore.so.4 (0x00007ff37ad0f000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff37aa0c000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff37a7f5000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff37a42c000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff37a20f000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ff379ff7000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff379df3000) libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007ff379af7000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff3798ee000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff3795e9000) /lib64/ld-linux-x86-64.so.2 (0x00007ff37b5e5000) libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007ff3793a9000)
Also, just to make sure, I tried nm on that content library and I can see the MD5 entry:
w _ITM_registerTMCloneTable00000000003c9468 d __JCR_END__00000000003c9468 d __JCR_LIST__ w _Jv_RegisterClasses U MD5 <---- it's here... U memcmp@@GLIBC_2.2.5 w pthread_cancel U pthread_mutex_destroy@@GLIBC_2.2.5
I try to reproduce the hash of prestashop's password with nodejs and I find a difference in php.
<?php define(_COOKIE_KEY_, 'foo');$pass = 'bar';$hash1 = md5(_COOKIE_KEY_ . $pass);$hash2 = md5('foo' . $pass);// $hash1 !== $hash2
In NodeJS (with md5 module) I found only the $hash2 but impossible to find $hash1.
Someone know why ? and how to reproduce ?
Thanks a lot.