Mathematica, 1067 bytes (Mac OS Roman character encoding)

FromCharacterCode[Mod[Tr/@{c=ToCharacterCode@#,c^2},216,32],"MacintoshRoman"]/.Inner[Rule,";¤7«´3πœ(ú-UU=6z±'¥ÿ†tƒ|\¢KÛd§≤jSóKWÊ8‰Ñwiøì¡ÛhÓ\‡¨:–*~‚¬æº¢»‘¤Á^∫„·nLÒ¤b|$|ÇòCóÌÈS_Ñä.Ëí 5y«KΔË\Ãò™_E`J’ëΔñTV–N„'„Ÿà¥xpîH#-PP)ÈÊVQ©LrBt}∑WÉ∏dÿå„•Tz∑Âao¿rÃ^bbP¨}ëÖ◇1èÇ&d¢¤ái√,B}±BˆÍdA´íBtæÅ/m√yQ6,uãÊ≤/Î!ïøuΩÒÉ)ë“∕C$RY•ÍÍu£oÉÓå‚Ïl.·1‚40ÃÚ¨ÇÆÅccflÓ8Ï Gáç3EÑ¥fXñ¨Àìz~j÷–ñÓz0~ôWtñ}μÎ◇f||Dd\ ÙH﷿É∑Ì´|¿Ö_»RT8Ûª|Äqü‘&6Ãác›Yˆ¿ô5≈ënÚqΩåVä>∫æ∂p ¨jtöåoÌfløÏÏò§¤flÈ;À∑Ѥ·›9né∕<·ì∕ÿmŸ«Ì»j√üà‰÷“5ïä^Ûe◇kd‡“(Ïö71›iΟÁm„ÈïÒß„kÕπ°ÊÓÒçÓfˆ¨flÁ9k|¶ä∕l~Òød‹jZÏ2[kÎ√3ÛâìÓΔE]ıIÚ>{#ÁÖ‚Üâ;·?l^vàß‹‘jîÙÇÅÉú¥äärÆæ™∏Üi≈mØÂ’-%USÌâ’ı Ê›·Ëÿb‡ıÖ31nh™Δ$~%À0n-À´sflk∑p.o5vz}mè]ÎÅç©lt;Îu„ŸW„›ˆˆÍ﷿Ä*7m8‰πór,„Õш/”Ë∕ªß9±‡¶çÁ•âg˜fló)ÖÔ¡'wúæ0ñ„Kûr"~(a=StringPartition)~2,"AAA&&DH&&&&&D&KKKKH&o&WT&&soV&bEYLDD:DDwDwDDDD&q&WBDyNNPuDj&FPhYss:c'ScAd:LdR&dvhhMZhE;MZv&MZHaEHve'evCdeHgSe:b ZaBa;mMrsZH'pGGMZGGo'NNDPuDFPgxNNdd&Hohoi&ufMZ&Tv:i&'pJdf;AafkMkZk;fdkm'iqlLFd:wtf&i&LhqhHYCnq&:jOODMoZooYf';&SCuwcSQiabrBDFNNrpT'qR:&Ysr eFDZmSajEvH'H'&ifHqtTUBVVF&du&ESnTdrdDugddd'nrWqXDyFYz"~a~1,List]/.x_/;StringLength@x>1->"@"&

Unnamed function taking a string as input and returning a character. The function has the following form:

1  FromCharacterCode[
2    Mod[Tr/@{c=ToCharacterCode@#,c^2},216,32]
3    ,"MacintoshRoman"] /.
4  Inner[Rule,
5    GIANT_STRING_1 ~(a=StringPartition)~2,
6    GIANT_STRING_2 ~a~1,
7    List]
8  /. x_/;StringLength@x>1 -> "@" &

Here GIANT_STRING_1 is a string containing 608 one-byte characters in the Mac OS Roman character encoding (none of which are in the range 00-1F), while GIANT_STRING_2 is a string containing 304 ASCII characters.

Line 2 starts the hash function: it converts the input string into a list of character codes (encoding irrelevant since they're all printable ASCII), then computes the sum of those character codes and the sum of their squares, both modulo 216 and forcing the answer to lie between 32 and 255. Then lines 1 and 3 convert those ordered pairs of integers into two-character strings, which is the hash value we ultimately use.

Line 5 turns GIANT_STRING_1 into a list of 304 two-character strings; line 6 turns GIANT_STRING_2 into a list of 304 one-character strings. Then lines 4 and 5 convert those two lists into a set of 304 replacement rules: if you see such-and-such two-character string, turn it into such-and-such one-character string. Finally, line 8 turns any remaining two-character string into "@".

There are 71 monsters in the list whose symbol is "@", and those are handled without hashing (I stole this idea from a comment by ais523 on another answer). It just so happens that the other 304 hash values are all unique! and so no other modifications to the algorithm are needed. (It's a lucky break that "human" needs to be mapped to "@", since the sums of the character codes of the letters in "human" and the letters in "shark" are identical, as are the sums of the squares of those codes—as integers, not even modulo 216!)

Java, 1130 bytes

import java.util.*;class G{public static void main(String[]u){BitSet s=BitSet.valueOf(Base64.getDecoder().decode("D94FuoCWYEIhCTEgLWwRNU/CMB1cE7XBhxBsBCusihaASRg14IJpQMOIDJdFx3BOdDcmThdhILVkCgGsEmhII8UE+SB4kDYEEJzw7Tw54oUEQZe0AUHCACH6nAdqgiZgJhASCIPAEAzJBmuMIrBCHE8IiFjgKQwrN4/90B4QFaLBQBEwTArRBMLCLHQOUQs7ZXZ8B8uGC1EbeAMJBdihUDgCIwGUEKgEAu4W2SJkIAhzB1IQSHgNiEAwABQECV5BvAB7eizABXxFLEg5iMA3whhAFXOKHXEURB7UA7PQjgUK7sji8CmIC0FJsTB4tAMFgiARB3hOJATDsBkgGKnGmWIiIWBRwkMgToQJ49G8gTR4IqcB4vJwDBHSVBLQhpwHsUFipqBcWWaEsCBoGBF0AlNAE305HAfdU1AEbELBO0EERAfkmMkgZcEXDIa4MAp4HcENmYAMBB7UBbTwBqQPSMS9kVkEBMhCudAqBAKaR1CzZggDRw8WMAh0FQPEyKAsRAxzBwn0grwDMQMyQMdADRtFUBAsBQetRRBwcUgrlsQ1IkosBc9B6iBcjAkSDDKgEAQ1wgLIMEEwMkYB42ERBCdiEJMAt1wYSIAQkdIEI0UPNhALsDnRQ1AT/HQi1AyCEwiICOICpiAPlB8MwxnBPIk6JYaIgDy8NJHDsiAqzK0JAXpQPXgPLwJuEEbMTAGBYlQbDESvAXJAAQ=="));int i,j,k,c,d,h=u[0].hashCode(),a=(h&4092)>>2|(h>>5)&1024|(h>>7)&2048|(h>>9)&4096;char r='@';for(i=k=0;i<4297;i+=14){for(c=0,j=7;j>=0;j--)c+=c+(s.get(i+j)?1:0);if((k+=c)==a){for(d=0,j=13;j>=8;j--)d+=d+(s.get(i+j)?1:0);r=d<5?" &':;".charAt(d):(char)((d<31?60:66)+d);}}System.out.println(r);}}

Ungolfed:

import java.util.*;

class G {
    public static void main(String[] u) {
        BitSet s = BitSet.valueOf(Base64.getDecoder().decode(
                "D94FuoCWYEIhCTEgLWwRNU/CMB1cE7XBhxBsBCusihaASRg14IJpQMOIDJdFx3BOdDcmThdhILVkCgGsEmhII8UE+SB4kDYEEJzw7Tw54oUEQZe0AUHCACH6nAdqgiZgJhASCIPAEAzJBmuMIrBCHE8IiFjgKQwrN4/90B4QFaLBQBEwTArRBMLCLHQOUQs7ZXZ8B8uGC1EbeAMJBdihUDgCIwGUEKgEAu4W2SJkIAhzB1IQSHgNiEAwABQECV5BvAB7eizABXxFLEg5iMA3whhAFXOKHXEURB7UA7PQjgUK7sji8CmIC0FJsTB4tAMFgiARB3hOJATDsBkgGKnGmWIiIWBRwkMgToQJ49G8gTR4IqcB4vJwDBHSVBLQhpwHsUFipqBcWWaEsCBoGBF0AlNAE305HAfdU1AEbELBO0EERAfkmMkgZcEXDIa4MAp4HcENmYAMBB7UBbTwBqQPSMS9kVkEBMhCudAqBAKaR1CzZggDRw8WMAh0FQPEyKAsRAxzBwn0grwDMQMyQMdADRtFUBAsBQetRRBwcUgrlsQ1IkosBc9B6iBcjAkSDDKgEAQ1wgLIMEEwMkYB42ERBCdiEJMAt1wYSIAQkdIEI0UPNhALsDnRQ1AT/HQi1AyCEwiICOICpiAPlB8MwxnBPIk6JYaIgDy8NJHDsiAqzK0JAXpQPXgPLwJuEEbMTAGBYlQbDESvAXJAAQ=="));

        int i, j, k, c, d, h = u[0].hashCode(), 
            a = (h & 4092) >> 2 | (h >> 5) & 1024 | (h >> 7) & 2048 | (h >> 9) & 4096;
        char r = '@';
        for (i = 0, k = 0; i < 4297; i += 14) {
            for (c = 0, j = 7; j >= 0; j--)
                c += c + (s.get(i + j) ? 1 : 0);
            if ((k += c) == a) {
                for (d = 0, j = 13; j >= 8; j--)
                    d += d + (s.get(i + j) ? 1 : 0);
                r = d < 5 ? " &':;".charAt(d) : (char) ((d < 31 ? 60 : 66) + d);
            }
        }
        System.out.println(r);
    }
}

Monster names are:

• hashed using Java hashcode method => 32 bits
• ANDed with mask 1001001000111111111100 => 13 bits
• sorted from smallest to biggest
• we then use the delta values of the sorted list => 8 bits

The display character is encoded on 6 bits.

So each tuple (monster name, character) uses 14 bits. All the tuples are saved in a BitSet and base 64 encoded.

I lose a lot of bytes with base64 encoding and BitSet operations :-)

Javascript, 1361 1152 bytes

b=>{for(a of "AAx5Al5An6cl6kn6_&As8Bb9Dh5DnaDr8Fe6Gn6Jx7MliNk6Os5PeaYu8bg6blcbladi6es6hu6hlcil9is7mnbmh8neapd9sn9ss8vk5wnb_DCngIh5bnhbngbngbnhbnibnfbnibnhbnibncbnbgnbgncondrnasndwncynd_HCs7Lrben5ftaftbgt5htamr8stbstbtn5_KKpcKnbKteKtc_oMcaUi8gn6hc8hn9onaoc3onb_WNl6btcwh6_TOi8il9rlatl5wlb_sSs8crbce9grcsn8_VVrgvdcve7_bab9gefqbe_Ealdelfflesr7wlf_Yae3cefmy6or7sh9yi4_Lah9dh8lh4mhb_:beecn9ce9go5ia6ld6nt4sra_wbmebmglm9pmb_qbmela9mn8mk5re5tebws6_Bbt3gt9rn5vtb_ybtbytc_Nbgkbaagglgabggngadrgira8_Pbgdbgdge9geb_ubndgnche5py4we8wnd_jbyaoybsyd_Fbdagdaln6rd8sr8vsdydb_hbr7dgaddadf5ht6mrimrb_ccebceapk8_'cmafmbgmbgmaimalmdpmbrmasmbsmbwma_Sca5gecpr9pn6se5wne_dce6do5dg3fx3hpehdajl6lg9lgawg4wlawf8wbfwfbwf4_Rdrcrrc_vdxbexdfxbfd9ixasxc_Mdybey9eybgybgybhybkycoy9_Zdeceeaeecgl5gecgecheckedoeasn8_;elcgl9jh9kn6pa7sk5_aft8gt9geckeaqe9stb_efeefecfefge9sef_Cfremrgpre_gge8gn7wef_ gt5se5_mgcblcbscb_rgt9rt9re9st9wt7wk9_pgrdircrrc_Gggagdage5gde_xgg8xn3_ihsaip3le6ms5qt6tu5_fht8jr6kn6lt9lx4pr7tr5_Jjka_kkdbkndkd6ldc_llna_tlectr7_nmhewhbwha_Oog9od9oe4_Qqcg_Uuka_Xxn4_zzy5".split("_"))if(~a.substring(1).indexOf(b[0]+b[b.length-1]+b.length.toString(36)))return a[0];return "@"};

Python, 2055 bytes

def f(s):import re;m=re.search(s[-3:-1]+s[:2]+str(len(s))+"(.)","omgn5Gteen13vligr7glero10'akga12Sanso11aragi9rgoDe10&tiet5HleVl16Vanst11Hmpmo14nubge15brsho5uingo21Nbuin7&gobl11Dgobl12Duaja6faule10lkest7Eolic9Ttawa15EanKo14KanKo12Kviba12&gore10Dimim3iutzr5zingn10Ganhi10Herfr15emmel9Mlele13'guNa6Wkaja6dotco6docvr5&petr7tmaor10oarli6:nhpi7;moma11&icli4Linbl20Nolwa11Titwr6Wtlgi12ateru12Rbign12Zozgr9Plepa11'oufo9vingu23Norhi8onena10&tati5Hiosc8sinre18Nligo6obeki10aeaow7Yeyfl12elewo10'adsh5 anfr11Hapap3Ygrog4Obequ9ahopy6Steki6fgogr11Dgogr12Dpepi9Sngdi5dindw10hlegl11'imgr11Pbava11Bcero12phaOl8Tdoli10dcuwi15dargn14GotIx5Dinbl13Parwa4dpuhe14dtisa9&ilba14:liho9onyer6&euAs8&aupl14Cttle9qmmdw11Molbr10Fmism11mncPe10&mpwa11noror3oispy8caumo16Clest11'haUr8okekr6;bigi12ZbuBa9&gowh12Dbiko13Zbiet12Zmmgn11Molwe8dmowa11&icde8Lbiho6hdola9dleJu7&otMi18&ulum10Uenpi9&luho10ighye12ymamu5qorwh13ughbl11yylga8gKoKe12Knndj6&mmet11Magbl10Narsh5;osgh5 orxo4Xoltr5Tdoma8qopCy7Hceir12pgoba18Dorlo9wgoba16Dbidw12ZinFa6&goor13DeaAl5Aiuba14qloac9bkemo6Yniqu16QteDi8&aufo14Ckesh8Fetye4Yolro9ryema18hersh15eaggo11Nrase9ranig6:ghba12Winbr13Polwi11dgeti5fzoNa6&orga9emmko12Manfi8aorgn10Gatco6Alecl10'goye13Deabu7hinog9Oheli6Feoch9:ynly4fngte5ieeel12;rawe7ricch11caior11ocala9fguvi13Fangi9aangi5Hhepa7fdesa10:cuOr5&rswa8ubrco5Sorva12Vxaxa3xovlu12tbaba3Bilcr9:geAn5Aolwo4dviic9&tafi14Ecegl13pbugr8xorpu11wgoCh16Dicar9Laggu13Ndegi12shoAr6Aolla12kedce9sitma8&erti11qicma11Lbior10Zviho12&test12vbusu8&fofo3ddeca11srara9rolko6kmpwo10ntaea15Ellbl10jgosi13Daksn5Svibo10&tosk8Zicco10cvera5Bgoba15DatDe5&goba17Dpuwu6qkawe10dmmhu11Mdodo3dunhe10dtcsa9Yckge5:tefi11vsiqu6iloqu14bewne4:yoGe6&caho8fucwo9rorMo10oisje9;taai13Eardw5holye11Fordw10hlloc11jough5Zerfl14emila11mtedu11vthro5qteic10vtuLo11Hmmor9Mirva7Vbagi9Bolro10Tmako13kleir10'biel10Zmmgi11Mnema5ilego10'olre8Forbl13usiwa14Sroba6&agre8Nrohe6&orgr12ulefl11'ocja10JghYe8&aumi8HiuSc8sbihu12Zriki6Ayemi11horko11kolgr10Furle6ianfi10Hmigi11monpo4ullsp13jaiKo11Ktedi12Rapca15Yorog9Oylwi15geegi9;orba14worba16w");return m.group(1)if m else'@'

Here's my test harness, in case it helps anyone else.

MAPPING = {
    # as given in the original question
}
def validate_solution(f):
    for k, v in MAPPING.iteritems():
        vv = f(k)
        if vv != v:
            print 'FAIL: f(%s) = %s instead of %s' % (k, vv, v)
    print 'SUCCESS!'

I wrote a little program to enumerate all the various ways of extracting 4 characters plus the length of the string. My original plan had been to then ord() these characters, do some math on them, and boil it down to a perfect hash function that produced indices into a table of outputs. So I wrote another little program to enumerate all the various ways of summing/multiplying/modulo'ing these 4 characters together; but the resulting hash functions kept having way too many collisions. So eventually I gave up and just did what you see here, which is just a map from the small-string representation of each monster's name to the appropriate symbol.

That is: What I wanted to get was

def f(monster_name):
    small_string = f1(monster_name)
    integer_index = f2(small_string)
    symbol = "relatively_short_table"[integer_index]
    return symbol

but I only managed to get as far as

def f(monster_name):
    small_string = f1(monster_name)
    symbol = {relatively_large_dict}[small_string]
    return symbol

where my dict lookup {relatively_large_dict}[small_string] is expressed as re.match(small_string+"(.)", "relatively_large_string") for golfiness.